I've been playing with Databricks Lakebase lately and I'm impressed by some of the features that come with it. I mean, you can sync data from a Lakehouse to Lakebase (OLAP to OLTP) and vice versa using Lakebase CDF. But most importantly, I'm impressed by how Lakebase branching works.
Traditionally, setting up a new environment meant performing a dump of the data and waiting for that to finish. This could take minutes or hours or days (depending on the size of the data). Moreover, the data becomes stale the moment you start copying it; hence, your pipeline is being tested with older data when performing a full copy. Setting access, managing resources for the full replication, and the cost for storing the data are some other factors that make developers think before they create a new env.
Lakebase solves that problem with database branching. Database branching, unlike database copy, points to the same storage without copying anything. That means if you create a dev branch from prod, they point to the same schema and data at a specific point in time, share the same underlying storage without data duplication and data is stored only when changes happen. This is called copy-on-write.
So why is this possible now? Why isn't traditional Postgres able to do this? In a traditional pg server, the compute and storage are tightly coupled. The database process and the data live in the same instance. Hence, the only option is to perform a full copy of the data.
What Databricks has done is decouple the storage and compute, allowing data to be written to a versioned storage engine. This means that lakebase versions each change instead of overwriting it! Same concept as Delta tables. That also unlocks a powerful tool! Time Travelling!
Now, you can have a branch for each developer, each PR, and each test run. You can also create and delete these branches at your own convenience!
Databricks has consistently been solving complex data engineering problems, and this is one of the big ones!
How do you use Lakebase in your projects? Excited to know!
