https://community.databricks.com/t5/community-articles/is-zerobus-the-future-of-ingestion-on-databricks-lessons-from-a/m-p/148909#M1020 <P>Between 2019 and 2021, we built a multi-market payments data platform on Databricks that now processes more than <STRONG>7 billion transactions per year</STRONG> across seven markets.</P><P>Ingestion was by far the most operationally complex layer.</P><P>To support MongoDB CDC streams, we engineered:</P><UL><LI><P>A custom Python CDC publisher</P></LI><LI><P>Azure Event Hubs as the message backbone</P></LI><LI><P>Avro landing in the raw layer</P></LI><LI><P>A generic Spark Structured Streaming framework to load Bronze (Delta)</P></LI></UL><P>The architecture worked and scaled — but it required significant custom engineering, careful orchestration, and continuous operational attention as data volume and dataset count increased.</P><P>Looking at the capabilities available today, especially Zerobus, it’s hard not to see how much simpler this ingestion layer could become. While still in preview, Zerobus represents a shift toward reducing message-bus dependency, custom streaming frameworks, and ingestion-specific infrastructure.</P><P>If it matures as expected, it has strong potential to become the default solution for near–real-time ingestion on Databricks.</P><P>I wrote a detailed breakdown of the original architecture, the scaling challenges we encountered, and why Zerobus may fundamentally change how ingestion is designed going forward.<BR /><BR /><span class="lia-unicode-emoji" title=":link:">🔗</span> <A href="https://medium.com/@wesley.felipe/databricks-lakehouse-without-the-workarounds-part-1-0d2101fb9f34" target="_blank" rel="noopener">[Medium]&nbsp;Databricks Lakehouse Without the Workarounds — Part 1: Ingestion</A><BR /><BR /></P> Fri, 20 Feb 2026 15:49:08 GMT wesleyfelipe 2026-02-20T15:49:08Z https://community.databricks.com/t5/community-articles/is-zerobus-the-future-of-ingestion-on-databricks-lessons-from-a/m-p/148909#M1020 <P>Between 2019 and 2021, we built a multi-market payments data platform on Databricks that now processes more than <STRONG>7 billion transactions per year</STRONG> across seven markets.</P><P>Ingestion was by far the most operationally complex layer.</P><P>To support MongoDB CDC streams, we engineered:</P><UL><LI><P>A custom Python CDC publisher</P></LI><LI><P>Azure Event Hubs as the message backbone</P></LI><LI><P>Avro landing in the raw layer</P></LI><LI><P>A generic Spark Structured Streaming framework to load Bronze (Delta)</P></LI></UL><P>The architecture worked and scaled — but it required significant custom engineering, careful orchestration, and continuous operational attention as data volume and dataset count increased.</P><P>Looking at the capabilities available today, especially Zerobus, it’s hard not to see how much simpler this ingestion layer could become. While still in preview, Zerobus represents a shift toward reducing message-bus dependency, custom streaming frameworks, and ingestion-specific infrastructure.</P><P>If it matures as expected, it has strong potential to become the default solution for near–real-time ingestion on Databricks.</P><P>I wrote a detailed breakdown of the original architecture, the scaling challenges we encountered, and why Zerobus may fundamentally change how ingestion is designed going forward.<BR /><BR /><span class="lia-unicode-emoji" title=":link:">🔗</span> <A href="https://medium.com/@wesley.felipe/databricks-lakehouse-without-the-workarounds-part-1-0d2101fb9f34" target="_blank" rel="noopener">[Medium]&nbsp;Databricks Lakehouse Without the Workarounds — Part 1: Ingestion</A><BR /><BR /></P> Fri, 20 Feb 2026 15:49:08 GMT https://community.databricks.com/t5/community-articles/is-zerobus-the-future-of-ingestion-on-databricks-lessons-from-a/m-p/148909#M1020 wesleyfelipe 2026-02-20T15:49:08Z