Zerobus to lakebase

[MikeGo]

Hi,

Is there a plan to make lakebase as destination of Zerobus?

Thanks.

[DB1To3]

Did you watch the Bricktalks from last week about zerobus?
https://community.databricks.com/t5/bricktalks-tv/recording-bricktalk-zerobus-ingest-amp-cdc-workflo...

Your question came up in the Q&A.

Vicky Bukta is the PM and based on what I heard, it didn't seem to be a priority.  I think they want you to curate/refine your data as you move it from a UC managed table into lakebase.

IMHO, I don't think it is reasonable to expect lakebase as a destination.  There are already a lot of considerations/complexity when landing data into open table formats, let alone conventional database tables.  I think it would be 10x more complex to make it work smoothly with lakebase, and even after it worked you would see a massively lower level of performance.  Conventional databases have lots of additional considerations, like managing indexes, and respecting pessimistic locks/isolation.  A UC managed table doing append-only additions can be extremely simple and fast.

[MikeGo]

Thanks for the link. IMHO, it is the opposite. Previously when there is no option, we have to load data from Kafka to delta table. Now with Lakebase and LTAP, we can consider to use Lakebase as a fast landing zone. Instead of loading data to Lakehouse first and then flow to Lakebase, we might land data to Lakebase first and flow back to Lakehouse. Or as advertised by Databricks, land to Lakebase only. With LTAP, it can be fast write to Lakebase and read it as iceberg or delta table.

[DB1To3]

The complexity I had mentioned would not be on your side of things.  It would be in the zerobus and its sink, rather than with the client-application.

Imagine a worst-case scenario where a customer puts 100 indexes on the lakebase table that is receiving data from zerobus, and then initates table-locks via SERIALIZABLE-isolation queries for 90% of the day.  I'm pretty sure the zerobus team does not want to deal with that sort of nonsense!

If that team is just writing to UC managed tables (so-called "writeserializable" isolation by default) then they can just drop files at full speed without worrying about the complexity that would come from a conventional OLTP database engine.  They can continue to assure their customers of the end-to-end latency of approximately 5 seconds.  But with lakebase there is no way they could assure anyone of 5 seconds of latency.  There are far more factors that are totally out of the control of Databricks, and every customer would experience the zerobus component differently.  At that point zerobus would just feel like a really, really crappy replacement for a kafka broker.  They need to make sure zerobus solves more problems than it causes, especially when compared to alternatives.

[MikeGo]

Why assume 100 indexes on a Lakebase landing table while picturing a Delta table with at most 4 LC keys? Nobody will create 100 indexes for a landing table.

Five seconds is far from a good num for some use cases. With Lakebase, data can be queried as soon as the transaction commits, potentially within milliseconds, while LTAP enables analytical access to the same logical data.