2 weeks ago
Hi,
Is there a plan to make lakebase as destination of Zerobus?
Thanks.
2 weeks ago
Did you watch the Bricktalks from last week about zerobus?
https://community.databricks.com/t5/bricktalks-tv/recording-bricktalk-zerobus-ingest-amp-cdc-workflo...
Your question came up in the Q&A.
Vicky Bukta is the PM and based on what I heard, it didn't seem to be a priority. I think they want you to curate/refine your data as you move it from a UC managed table into lakebase.
IMHO, I don't think it is reasonable to expect lakebase as a destination. There are already a lot of considerations/complexity when landing data into open table formats, let alone conventional database tables. I think it would be 10x more complex to make it work smoothly with lakebase, and even after it worked you would see a massively lower level of performance. Conventional databases have lots of additional considerations, like managing indexes, and respecting pessimistic locks/isolation. A UC managed table doing append-only additions can be extremely simple and fast.
a week ago
Thanks for the link. IMHO, it is the opposite. Previously when there is no option, we have to load data from Kafka to delta table. Now with Lakebase and LTAP, we can consider to use Lakebase as a fast landing zone. Instead of loading data to Lakehouse first and then flow to Lakebase, we might land data to Lakebase first and flow back to Lakehouse. Or as advertised by Databricks, land to Lakebase only. With LTAP, it can be fast write to Lakebase and read it as iceberg or delta table.
a week ago
The complexity I had mentioned would not be on your side of things. It would be in the zerobus and its sink, rather than with the client-application.
Imagine a worst-case scenario where a customer puts 100 indexes on the lakebase table that is receiving data from zerobus, and then initates table-locks via SERIALIZABLE-isolation queries for 90% of the day. I'm pretty sure the zerobus team does not want to deal with that sort of nonsense!
If that team is just writing to UC managed tables (so-called "writeserializable" isolation by default) then they can just drop files at full speed without worrying about the complexity that would come from a conventional OLTP database engine. They can continue to assure their customers of the end-to-end latency of approximately 5 seconds. But with lakebase there is no way they could assure anyone of 5 seconds of latency. There are far more factors that are totally out of the control of Databricks, and every customer would experience the zerobus component differently. At that point zerobus would just feel like a really, really crappy replacement for a kafka broker. They need to make sure zerobus solves more problems than it causes, especially when compared to alternatives.
a week ago
Why assume 100 indexes on a Lakebase landing table while picturing a Delta table with at most 4 LC keys? Nobody will create 100 indexes for a landing table.
Five seconds is far from a good num for some use cases. With Lakebase, data can be queried as soon as the transaction commits, potentially within milliseconds, while LTAP enables analytical access to the same logical data.
an hour ago
I can't speak to roadmap commitments in a public forum, and as far as I know Lakebase isn't a direct Zerobus Ingest target today. Your account team can tell you whether it's planned and under what NDA terms.
What works today
Zerobus Ingest is designed to write records directly into Unity Catalog Delta tables. Lakebase is managed Postgres for OLTP-style access, so the supported pattern is two hops:
Choosing a sync mode
If you need the data in Lakebase quickly, use a continuous sync. The source Delta table needs Change Data Feed enabled for this:
ALTER TABLE my_catalog.my_schema.events
SET TBLPROPERTIES (delta.enableChangeDataFeed = true);
If a few minutes or more of lag is fine, a triggered sync is cheaper.
A snapshot sync suits small or slowly changing tables.
Why the two hops are usually fine
The Delta table is your system of record for analytics, ML, and governance. Lakebase serves the low-latency lookups and application reads on top of it. Because the sync runs on managed infrastructure, you don't have to build and operate your own writer to Postgres.
If you need to write to Postgres directly
If you need events to land in Postgres with no Delta table involved, for example for transactional write-path semantics, Zerobus isn't the right tool. Write to Lakebase directly with a standard Postgres client or driver from your application.
If you tell me your latency target and whether you also need the data in Delta, I can suggest which of these patterns fits best.