<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic Re: Zerobus to lakebase in Data Engineering</title>
    <link>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/177165#M57258</link>
    <description>&lt;DIV&gt;
&lt;P&gt;Yes, that pattern works, and Zerobus isn't needed for it. Put Kafka between OTel and Lakebase as the buffer, then have a consumer write into Lakebase over its Postgres endpoint. Nothing needs to land in Delta first.&lt;/P&gt;
&lt;P&gt;I'm assuming LTAP means the Lakebase-to-lakehouse analytics path. If you mean something else, tell me and I'll adjust.&lt;/P&gt;
&lt;P&gt;&lt;STRONG&gt;Recommended flow&lt;/STRONG&gt;&lt;/P&gt;
&lt;OL&gt;
&lt;LI&gt;&lt;STRONG&gt;OTel Collector → Kafka.&lt;/STRONG&gt; The collector's Kafka exporter is fire-and-forget from the app's point of view, and Kafka absorbs bursts and gives you replay.&lt;/LI&gt;
&lt;LI&gt;&lt;STRONG&gt;Kafka → Lakebase.&lt;/STRONG&gt; You have two options:
&lt;UL&gt;
&lt;LI&gt;&lt;STRONG&gt;A Databricks streaming job.&lt;/STRONG&gt; Structured Streaming or a Lakeflow pipeline with the Kafka source [1] gives you checkpointing and managed retries. Use &lt;CODE&gt;foreachBatch&lt;/CODE&gt; to write each micro-batch to Lakebase over Postgres.&lt;/LI&gt;
&lt;LI&gt;&lt;STRONG&gt;A consumer outside Databricks.&lt;/STRONG&gt; A Kafka Connect JDBC sink or a small service that writes straight to the Lakebase Postgres endpoint. Pick this if you don't want a Databricks cluster running just to move data.&lt;/LI&gt;
&lt;/UL&gt;
&lt;/LI&gt;
&lt;LI&gt;&lt;STRONG&gt;Lakebase → analytics.&lt;/STRONG&gt; Use whatever Lakebase-to-lakehouse path your LTAP plans rely on, downstream of this.&lt;/LI&gt;
&lt;/OL&gt;
&lt;P&gt;&lt;STRONG&gt;Sketch of the &lt;CODE&gt;foreachBatch&lt;/CODE&gt; option&lt;/STRONG&gt;&lt;/P&gt;
&lt;PRE&gt;&lt;CODE class="language-python"&gt;def write_to_lakebase(batch_df, batch_id):
    # 1. Bulk-write the micro-batch into a staging table in Lakebase (JDBC/Postgres)
    (batch_df.write
        .format("jdbc")
        .option("url", jdbc_url)              # Lakebase Postgres endpoint
        .option("dbtable", "staging.otel_events")
        .option("user", user)
        .option("password", oauth_token)      # short-lived token, refresh it
        .mode("overwrite")
        .save())
    # 2. Merge into the target idempotently, e.g. via psycopg:
    #    INSERT INTO otel_events SELECT * FROM staging.otel_events
    #    ON CONFLICT (event_id) DO NOTHING;

(spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", brokers)
    .option("subscribe", "otel-events")
    .load()
    # parse / flatten OTel payload here
    .writeStream
    .foreachBatch(write_to_lakebase)
    .option("checkpointLocation", checkpoint_path)
    .start())
&lt;/CODE&gt;&lt;/PRE&gt;
&lt;P&gt;&lt;STRONG&gt;Things that matter in practice&lt;/STRONG&gt;&lt;/P&gt;
&lt;UL&gt;
&lt;LI&gt;&lt;STRONG&gt;Make writes idempotent.&lt;/STRONG&gt; Kafka to Postgres is at-least-once, because a batch can be replayed after a failure. Give each event a stable key and use &lt;CODE&gt;INSERT ... ON CONFLICT DO NOTHING&lt;/CODE&gt; (or &lt;CODE&gt;DO UPDATE&lt;/CODE&gt;) so replays are harmless.&lt;/LI&gt;
&lt;LI&gt;&lt;STRONG&gt;Batch your writes.&lt;/STRONG&gt; Use micro-batches with a staging table or &lt;CODE&gt;COPY&lt;/CODE&gt;, not row-at-a-time inserts. That's the biggest throughput lever on the Postgres side.&lt;/LI&gt;
&lt;LI&gt;&lt;STRONG&gt;Handle credentials.&lt;/STRONG&gt; Lakebase connections use short-lived OAuth tokens, so make sure long-running writers refresh them rather than caching one for the life of the job.&lt;/LI&gt;
&lt;LI&gt;&lt;STRONG&gt;Size for telemetry volume.&lt;/STRONG&gt; Raw OTel traces and logs can be very high-volume and append-heavy, which Postgres handles less cheaply than Delta. Land in Lakebase only what needs low-latency or transactional access, such as recent state, aggregates, or lookups. Keep the full-fidelity firehose in Delta via the Kafka ingestion path [1].&lt;/LI&gt;
&lt;/UL&gt;
&lt;P&gt;If you share your expected events per second and what the app reads from Lakebase, I can suggest whether to use the in-Databricks streaming writer or an external sink.&lt;/P&gt;
&lt;H2&gt;References&lt;/H2&gt;
&lt;P&gt;[1] Ingest with SQL and Spark APIs | Databricks on AWS — &lt;A href="https://docs.databricks.com/aws/en/ingestion" target="_blank"&gt;https://docs.databricks.com/aws/en/ingestion&lt;/A&gt;&lt;/P&gt;
&lt;/DIV&gt;</description>
    <pubDate>Sat, 21 Nov 2026 05:34:02 GMT</pubDate>
    <dc:creator>anuj_lathi</dc:creator>
    <dc:date>2026-11-21T05:34:02Z</dc:date>
    <item>
      <title>Zerobus to lakebase</title>
      <link>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/175997#M57085</link>
      <description>&lt;P&gt;Hi,&lt;/P&gt;&lt;P&gt;Is there a plan to make lakebase as destination of Zerobus?&lt;/P&gt;&lt;P&gt;Thanks.&lt;/P&gt;</description>
      <pubDate>Mon, 09 Nov 2026 18:36:37 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/175997#M57085</guid>
      <dc:creator>MikeGo</dc:creator>
      <dc:date>2026-11-09T18:36:37Z</dc:date>
    </item>
    <item>
      <title>Re: Zerobus to lakebase</title>
      <link>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/176000#M57086</link>
      <description>&lt;P&gt;Did you watch the Bricktalks from last week about zerobus?&lt;BR /&gt;&lt;A href="https://community.databricks.com/t5/bricktalks-tv/recording-bricktalk-zerobus-ingest-amp-cdc-workflows/ba-p/175761" target="_blank"&gt;https://community.databricks.com/t5/bricktalks-tv/recording-bricktalk-zerobus-ingest-amp-cdc-workflows/ba-p/175761&lt;/A&gt;&lt;BR /&gt;&lt;BR /&gt;Your question came up in the Q&amp;amp;A.&lt;/P&gt;&lt;P&gt;Vicky Bukta is the PM and based on what I heard, it didn't seem to be a priority.&amp;nbsp; I think they want you to curate/refine your data as you move it from a UC managed table into lakebase.&lt;/P&gt;&lt;P&gt;IMHO, I don't think it is reasonable to expect lakebase as a destination.&amp;nbsp; There are already a lot of considerations/complexity when landing data into open table formats, let alone conventional database tables.&amp;nbsp; I think it would be 10x more complex to make it work smoothly with lakebase, and even after it worked you would see a massively lower level of performance.&amp;nbsp; Conventional databases have lots of additional considerations, like managing indexes, and respecting pessimistic locks/isolation.&amp;nbsp; A UC managed table doing append-only additions can be extremely simple and fast.&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Mon, 09 Nov 2026 19:02:19 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/176000#M57086</guid>
      <dc:creator>DB1To3</dc:creator>
      <dc:date>2026-11-09T19:02:19Z</dc:date>
    </item>
    <item>
      <title>Re: Zerobus to lakebase</title>
      <link>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/176130#M57095</link>
      <description>&lt;P&gt;Thanks for the link. IMHO, it is the opposite. Previously when there is no option, we have to load data from Kafka to delta table. Now with Lakebase and LTAP, we can consider to use Lakebase as a fast landing zone. Instead of loading data to Lakehouse first and then flow to Lakebase, we might land data to Lakebase first and flow back to Lakehouse. Or as advertised by Databricks, land to Lakebase only. With LTAP, it can be fast write to Lakebase and read it as iceberg or delta table.&lt;/P&gt;</description>
      <pubDate>Wed, 11 Nov 2026 00:52:26 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/176130#M57095</guid>
      <dc:creator>MikeGo</dc:creator>
      <dc:date>2026-11-11T00:52:26Z</dc:date>
    </item>
    <item>
      <title>Re: Zerobus to lakebase</title>
      <link>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/176175#M57101</link>
      <description>&lt;P&gt;The complexity I had mentioned would not be on your side of things.&amp;nbsp; It would be in the zerobus and its sink, rather than with the client-application.&lt;/P&gt;&lt;P&gt;Imagine a worst-case scenario where a customer puts 100 indexes on the lakebase table that is receiving data from zerobus, and then initates table-locks via SERIALIZABLE-isolation queries for 90% of the day.&amp;nbsp; I'm pretty sure the zerobus team does not want to deal with that sort of nonsense!&lt;/P&gt;&lt;P&gt;If that team is just writing to UC managed tables (so-called "writeserializable" isolation by default) then they can just drop files at full speed without worrying about the complexity that would come from a conventional OLTP database engine.&amp;nbsp; They can continue to assure their customers of the end-to-end latency of approximately 5 seconds.&amp;nbsp; But with lakebase there is no way they could assure anyone of 5 seconds of latency.&amp;nbsp; There are far more factors that are totally out of the control of Databricks, and every customer would experience the zerobus component differently.&amp;nbsp; At that point zerobus would just feel like a really, really crappy replacement for a kafka broker.&amp;nbsp; They need to make sure zerobus solves more problems than it causes, especially when compared to alternatives.&lt;/P&gt;</description>
      <pubDate>Wed, 11 Nov 2026 12:33:48 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/176175#M57101</guid>
      <dc:creator>DB1To3</dc:creator>
      <dc:date>2026-11-11T12:33:48Z</dc:date>
    </item>
    <item>
      <title>Re: Zerobus to lakebase</title>
      <link>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/176200#M57107</link>
      <description>&lt;P&gt;Why assume 100 indexes on a Lakebase landing table while picturing a Delta table with at most 4 LC keys? Nobody will create 100 indexes for a landing table.&lt;/P&gt;&lt;P&gt;Five seconds is far from a good num for some use cases. With Lakebase, data can be queried as soon as the transaction commits, potentially within milliseconds, while LTAP enables analytical access to the same logical data.&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Wed, 11 Nov 2026 19:11:35 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/176200#M57107</guid>
      <dc:creator>MikeGo</dc:creator>
      <dc:date>2026-11-11T19:11:35Z</dc:date>
    </item>
    <item>
      <title>Re: Zerobus to lakebase</title>
      <link>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/177152#M57251</link>
      <description>&lt;DIV&gt;
&lt;P&gt;I can't speak to roadmap commitments in a public forum, and as far as I know Lakebase isn't a direct Zerobus Ingest target today. Your account team can tell you whether it's planned and under what NDA terms.&lt;/P&gt;
&lt;P&gt;&lt;STRONG&gt;What works today&lt;/STRONG&gt;&lt;/P&gt;
&lt;P&gt;Zerobus Ingest is designed to write records directly into Unity Catalog Delta tables. Lakebase is managed Postgres for OLTP-style access, so the supported pattern is two hops:&lt;/P&gt;
&lt;OL&gt;
&lt;LI&gt;&lt;STRONG&gt;Zerobus → Unity Catalog Delta table.&lt;/STRONG&gt; Your producers push events straight into the Delta table with low latency and no message bus in between.&lt;/LI&gt;
&lt;LI&gt;&lt;STRONG&gt;Delta table → Lakebase via a synced table.&lt;/STRONG&gt; A synced table replicates the Delta table into a Postgres table in your Lakebase instance, so apps and services can query it with normal Postgres semantics.&lt;/LI&gt;
&lt;/OL&gt;
&lt;P&gt;&lt;STRONG&gt;Choosing a sync mode&lt;/STRONG&gt;&lt;/P&gt;
&lt;UL&gt;
&lt;LI&gt;
&lt;P&gt;If you need the data in Lakebase quickly, use a continuous sync. The source Delta table needs Change Data Feed enabled for this:&lt;/P&gt;
&lt;PRE&gt;&lt;CODE class="language-sql"&gt;ALTER TABLE my_catalog.my_schema.events
SET TBLPROPERTIES (delta.enableChangeDataFeed = true);
&lt;/CODE&gt;&lt;/PRE&gt;
&lt;/LI&gt;
&lt;LI&gt;
&lt;P&gt;If a few minutes or more of lag is fine, a triggered sync is cheaper.&lt;/P&gt;
&lt;/LI&gt;
&lt;LI&gt;
&lt;P&gt;A snapshot sync suits small or slowly changing tables.&lt;/P&gt;
&lt;/LI&gt;
&lt;/UL&gt;
&lt;P&gt;&lt;STRONG&gt;Why the two hops are usually fine&lt;/STRONG&gt;&lt;/P&gt;
&lt;P&gt;The Delta table is your system of record for analytics, ML, and governance. Lakebase serves the low-latency lookups and application reads on top of it. Because the sync runs on managed infrastructure, you don't have to build and operate your own writer to Postgres.&lt;/P&gt;
&lt;P&gt;&lt;STRONG&gt;If you need to write to Postgres directly&lt;/STRONG&gt;&lt;/P&gt;
&lt;P&gt;If you need events to land in Postgres with no Delta table involved, for example for transactional write-path semantics, Zerobus isn't the right tool. Write to Lakebase directly with a standard Postgres client or driver from your application.&lt;/P&gt;
&lt;P&gt;If you tell me your latency target and whether you also need the data in Delta, I can suggest which of these patterns fits best.&lt;/P&gt;
&lt;/DIV&gt;</description>
      <pubDate>Fri, 20 Nov 2026 19:34:00 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/177152#M57251</guid>
      <dc:creator>anuj_lathi</dc:creator>
      <dc:date>2026-11-20T19:34:00Z</dc:date>
    </item>
    <item>
      <title>Re: Zerobus to lakebase</title>
      <link>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/177159#M57254</link>
      <description>&lt;P&gt;Thanks. I don't want to go Delta table -&amp;gt; Sync. Instead I want to do the opposite. Quickly land on Lakebase and then likely take advantage of LTAP.&amp;nbsp;&lt;BR /&gt;Sources e.g. OTel can send data to cache and forget it, and then flow to Lakebase async and quickly. And yes, we can use Kafka.&lt;/P&gt;</description>
      <pubDate>Fri, 20 Nov 2026 21:51:04 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/177159#M57254</guid>
      <dc:creator>MikeGo</dc:creator>
      <dc:date>2026-11-20T21:51:04Z</dc:date>
    </item>
    <item>
      <title>Re: Zerobus to lakebase</title>
      <link>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/177165#M57258</link>
      <description>&lt;DIV&gt;
&lt;P&gt;Yes, that pattern works, and Zerobus isn't needed for it. Put Kafka between OTel and Lakebase as the buffer, then have a consumer write into Lakebase over its Postgres endpoint. Nothing needs to land in Delta first.&lt;/P&gt;
&lt;P&gt;I'm assuming LTAP means the Lakebase-to-lakehouse analytics path. If you mean something else, tell me and I'll adjust.&lt;/P&gt;
&lt;P&gt;&lt;STRONG&gt;Recommended flow&lt;/STRONG&gt;&lt;/P&gt;
&lt;OL&gt;
&lt;LI&gt;&lt;STRONG&gt;OTel Collector → Kafka.&lt;/STRONG&gt; The collector's Kafka exporter is fire-and-forget from the app's point of view, and Kafka absorbs bursts and gives you replay.&lt;/LI&gt;
&lt;LI&gt;&lt;STRONG&gt;Kafka → Lakebase.&lt;/STRONG&gt; You have two options:
&lt;UL&gt;
&lt;LI&gt;&lt;STRONG&gt;A Databricks streaming job.&lt;/STRONG&gt; Structured Streaming or a Lakeflow pipeline with the Kafka source [1] gives you checkpointing and managed retries. Use &lt;CODE&gt;foreachBatch&lt;/CODE&gt; to write each micro-batch to Lakebase over Postgres.&lt;/LI&gt;
&lt;LI&gt;&lt;STRONG&gt;A consumer outside Databricks.&lt;/STRONG&gt; A Kafka Connect JDBC sink or a small service that writes straight to the Lakebase Postgres endpoint. Pick this if you don't want a Databricks cluster running just to move data.&lt;/LI&gt;
&lt;/UL&gt;
&lt;/LI&gt;
&lt;LI&gt;&lt;STRONG&gt;Lakebase → analytics.&lt;/STRONG&gt; Use whatever Lakebase-to-lakehouse path your LTAP plans rely on, downstream of this.&lt;/LI&gt;
&lt;/OL&gt;
&lt;P&gt;&lt;STRONG&gt;Sketch of the &lt;CODE&gt;foreachBatch&lt;/CODE&gt; option&lt;/STRONG&gt;&lt;/P&gt;
&lt;PRE&gt;&lt;CODE class="language-python"&gt;def write_to_lakebase(batch_df, batch_id):
    # 1. Bulk-write the micro-batch into a staging table in Lakebase (JDBC/Postgres)
    (batch_df.write
        .format("jdbc")
        .option("url", jdbc_url)              # Lakebase Postgres endpoint
        .option("dbtable", "staging.otel_events")
        .option("user", user)
        .option("password", oauth_token)      # short-lived token, refresh it
        .mode("overwrite")
        .save())
    # 2. Merge into the target idempotently, e.g. via psycopg:
    #    INSERT INTO otel_events SELECT * FROM staging.otel_events
    #    ON CONFLICT (event_id) DO NOTHING;

(spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", brokers)
    .option("subscribe", "otel-events")
    .load()
    # parse / flatten OTel payload here
    .writeStream
    .foreachBatch(write_to_lakebase)
    .option("checkpointLocation", checkpoint_path)
    .start())
&lt;/CODE&gt;&lt;/PRE&gt;
&lt;P&gt;&lt;STRONG&gt;Things that matter in practice&lt;/STRONG&gt;&lt;/P&gt;
&lt;UL&gt;
&lt;LI&gt;&lt;STRONG&gt;Make writes idempotent.&lt;/STRONG&gt; Kafka to Postgres is at-least-once, because a batch can be replayed after a failure. Give each event a stable key and use &lt;CODE&gt;INSERT ... ON CONFLICT DO NOTHING&lt;/CODE&gt; (or &lt;CODE&gt;DO UPDATE&lt;/CODE&gt;) so replays are harmless.&lt;/LI&gt;
&lt;LI&gt;&lt;STRONG&gt;Batch your writes.&lt;/STRONG&gt; Use micro-batches with a staging table or &lt;CODE&gt;COPY&lt;/CODE&gt;, not row-at-a-time inserts. That's the biggest throughput lever on the Postgres side.&lt;/LI&gt;
&lt;LI&gt;&lt;STRONG&gt;Handle credentials.&lt;/STRONG&gt; Lakebase connections use short-lived OAuth tokens, so make sure long-running writers refresh them rather than caching one for the life of the job.&lt;/LI&gt;
&lt;LI&gt;&lt;STRONG&gt;Size for telemetry volume.&lt;/STRONG&gt; Raw OTel traces and logs can be very high-volume and append-heavy, which Postgres handles less cheaply than Delta. Land in Lakebase only what needs low-latency or transactional access, such as recent state, aggregates, or lookups. Keep the full-fidelity firehose in Delta via the Kafka ingestion path [1].&lt;/LI&gt;
&lt;/UL&gt;
&lt;P&gt;If you share your expected events per second and what the app reads from Lakebase, I can suggest whether to use the in-Databricks streaming writer or an external sink.&lt;/P&gt;
&lt;H2&gt;References&lt;/H2&gt;
&lt;P&gt;[1] Ingest with SQL and Spark APIs | Databricks on AWS — &lt;A href="https://docs.databricks.com/aws/en/ingestion" target="_blank"&gt;https://docs.databricks.com/aws/en/ingestion&lt;/A&gt;&lt;/P&gt;
&lt;/DIV&gt;</description>
      <pubDate>Sat, 21 Nov 2026 05:34:02 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/177165#M57258</guid>
      <dc:creator>anuj_lathi</dc:creator>
      <dc:date>2026-11-21T05:34:02Z</dc:date>
    </item>
    <item>
      <title>Re: Zerobus to lakebase</title>
      <link>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/177169#M57259</link>
      <description>&lt;P&gt;By LTAP, I mean&amp;nbsp;&lt;A href="https://docs.databricks.com/aws/en/oltp/projects/ltap-overview" target="_blank"&gt;https://docs.databricks.com/aws/en/oltp/projects/ltap-overview&lt;/A&gt;.&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Sat, 21 Nov 2026 06:55:57 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/zerobus-to-lakebase/m-p/177169#M57259</guid>
      <dc:creator>MikeGo</dc:creator>
      <dc:date>2026-11-21T06:55:57Z</dc:date>
    </item>
  </channel>
</rss>

