article [PARTNER BLOG] Zerobus Ingest on Databricks in Technical Blog

article [PARTNER BLOG] Zerobus Ingest on Databricks in Technical Blog https://community.databricks.com/t5/technical-blog/partner-blog-zerobus-ingest-on-databricks/ba-p/142433 <P><LI-TOC indent="15" liststyle="square" maxheadinglevel="2"></LI-TOC></P> <H2> </H2> <H2><FONT size="5">Introduction</FONT></H2> <P><STRONG>TL;DR</STRONG></P> <BLOCKQUOTE class="graf graf--blockquote">ZeroBus Ingest is a serverless, Kafka-free ingestion service in Databricks that allows applications and IoT devices to stream data directly into Delta Lake with low latency and minimal operational overhead.</BLOCKQUOTE> <P>Real-time data ingestion is a core requirement for modern IoT and event-driven architectures. Traditionally, platforms like Apache Kafka have been used as an intermediary layer between producers and analytics systems, adding operational complexity and latency.</P> <P><STRONG>Zerobus Ingest is Databricks’ Kafka-free ingestion solution</STRONG> that allows applications and devices to write events directly into Delta tables with low latency and minimal infrastructure. In this article, we explore how Zerobus works, when to use it, and how to ingest real-time events step by step.</P> <P><STRONG>Zerobus</STRONG> <STRONG>connector</STRONG> Producers simply push data using a lightweight API, and Zerobus takes care of buffering, reliability, and scaling behind the scenes.</P> <H2><FONT size="5">The Challenge with Traditional Streaming Architectures</FONT></H2> <P>Traditional real-time data pipelines often rely on messaging systems like Kafka to move data from applications. While effective, this approach introduces several challenges, such as:</P> <UL> <LI><STRONG>Architecture Complexity:</STRONG> The introduction of a message bus adds extra layers to the data pipeline, making the overall architecture more complex and harder to maintain.</LI> <LI><STRONG>Operational Overhead:</STRONG> Managing and operating streaming infrastructure requires continuous effort across scaling, monitoring, security, and fault handling.</LI> <LI><STRONG>Increased Latency:</STRONG> Multiple handoffs between systems introduce delays, limiting the ability to deliver near real-time insights.</LI> </UL> <H2><SPAN>How Zerobus Ingest Enables Real-Time Ingestion in Databricks</SPAN></H2> <P><SPAN>Zerobus solves and simplify the ingestion process by: </SPAN></P> <UL> <LI><STRONG>Simplified architecture:</STRONG> Eliminates the need for a message broker by ingesting events directly into Delta Lake.</LI> <LI><STRONG>Lower operational overhead:</STRONG> Fully managed and serverless ingestion removes the burden of running and scaling streaming infrastructure.</LI> <LI><STRONG>Faster time to insights:</STRONG> Low-latency ingestion makes event data immediately available for analytics and reporting in Databricks.</LI> </UL> <P>In practice, Zerobus Ingest is designed for <STRONG>event-driven and IoT ingestion use cases</STRONG> where Databricks is the primary analytics platform.</P> <H2>Why Zerobus Matters for Modern Data Architectures</H2> <OL> <LI><STRONG>Eliminates the message bus layer (Kafka/Kinesis)</STRONG>, reducing pipeline hops and enabling faster <STRONG>real-time data ingestion on Databricks</STRONG>.</LI> <LI>Provides <STRONG>near real-time streaming ingestion</STRONG> with low operational overhead, making it simpler and more cost-efficient for <STRONG>event-driven architectures</STRONG>.</LI> </OL> <P>Modern data architectures increasingly favor <STRONG>simpler, event-first designs</STRONG> that minimize moving parts while preserving reliability and scale. <STRONG>Zerobus Ingest on Databricks</STRONG> supports this shift by reducing ingestion pipelines to their essential components and removing unnecessary infrastructure between event producers and <STRONG>Delta Lake storage</STRONG>. At the same time, the native integration with <STRONG>Unity Catalog</STRONG> ensures governance, security, and lineage are applied from the moment data is written.</P> <P>In short, Zerobus is a strong fit when <STRONG>Databricks is the primary destination</STRONG> and the goal is <STRONG>Kafka-free streaming ingestion into Delta Lake</STRONG>.</P> <H2>Key Component and Architecture : </H2> <P>Zerobus Ingest is implemented as a <STRONG>serverless ingestion layer within Databricks</STRONG>, exposed through gRPC and REST interfaces. Producers establish a streaming connection to the Zerobus endpoint and send events commonly serialised using Protocol Buffers directly to the target Delta table</P> <P>Behind the scenes, Zerobus integrates natively with <STRONG>Delta Lake</STRONG> for durable, transactional storage and <STRONG>Unity Catalog</STRONG> for access control and governance.</P> <P> Client SDKs in languages such as Python, Java, and Rust abstract the complexity of stream creation and record ingestion, allowing developers to implement scalable real-time pipelines with minimal configuration.</P> <P> <span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="anujsen18_0-1766508538746.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22503i39C47D6631256594/image-size/large?v=v2&px=999" role="button" title="anujsen18_0-1766508538746.png" alt="anujsen18_0-1766508538746.png" /></span></P> <P> </P> <P> </P> <P><STRONG>gRPC</STRONG>: A high-performance communication protocol used by Zerobus to stream events from producers to Databricks with low latency and reliability. <BR /><STRONG>Protobuf:</STRONG> A compact, strongly typed data format that defines the event schema and efficiently serializes data before ingestion. <BR /><STRONG>Client:</STRONG> Runs in the application or device, packages events using Protobuf, and sends them to Zerobus using the gRPC API. <BR /><STRONG>Zerobus Server:</STRONG> A fully managed, serverless Databricks service that receives events, handles buffering and durability, and writes data directly into Delta tables.</P> <H2> </H2> <H2>Zerobus vs Kafka vs Auto Loader: Databricks Ingestion Comparison</H2> <P>Zerobus Ingest <STRONG>complements, rather than replaces</STRONG>, existing Databricks ingestion tools. Each ingestion option is designed for a different data source type, latency requirement, and operational model. Choosing the right ingestion pattern depends on whether data is produced as events, files, or database changes, as well as the level of real-time processing and infrastructure complexity required.</P> <P>The following comparison summarises how <STRONG>Zerobus Ingest compares with Kafka, Auto Loader, and CDC pipelines in Databricks</STRONG>, and provides practical guidance on when each option is the best fit for real-time data ingestion workloads.</P> <P> </P> <TABLE border="1" width="100%"> <TBODY> <TR> <TD width="50%"><STRONG>Zerobus Ingest</STRONG><BR />Best for: Direct real-time event ingestion<BR />Data source type: Applications, IoT devices, services<BR />Latency: Low (near real-time)<BR />Operational effort: Very low (serverless)<BR />When to choose: When events need to land directly in Delta tables with minimal infrastructure and Databricks is the primary destination</TD> <TD width="50%"><STRONG>Auto Loader</STRONG><BR />Best for: Incremental file ingestion<BR />Data source type: Files in cloud storage<BR />Latency: Medium (micro-batch)<BR />Operational effort: Low<BR />When to choose: When data arrives as files or batches and near-real-time processing is not required</TD> </TR> <TR> <TD width="50%"><STRONG>Kafka + Structured Streaming</STRONG><BR />Best for: Large-scale event streaming and complex processing<BR />Data source type: Event streams via a message broker<BR />Latency: Low<BR />Operational effort: High<BR />When to choose: When multiple consumers, message retention, or advanced stream processing is required</TD> <TD width="50%"> <STRONG>CDC Pipelines</STRONG><BR />Best for: Database change data capture (CDC)<BR />Data source type: Transactional databases<BR />Latency: Medium to low<BR />Operational effort: Medium<BR />When to choose: When replicating database changes into the Lakehouse while maintaining row-level consistency</TD> </TR> </TBODY> </TABLE> <P><FONT size="4"><STRONG>Ingestion Method Decision Tree</STRONG></FONT></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="anujsen18_6-1766504369845.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22494i4019FA0950984840/image-size/medium?v=v2&px=400" role="button" title="anujsen18_6-1766504369845.png" alt="anujsen18_6-1766504369845.png" /></span></P> <P> </P> <H2>Ingesting Real time Messages using Zerobus Ingest </H2> <P><STRONG>Prerequisites & Environment Setup</STRONG></P> <P><STRONG>Server (Workspace side)</STRONG></P> <OL> <LI><STRONG> Zerobus Access:</STRONG> At the time of writing this article zerobus ingest is in public preview so if not enabled in your workspace please connect your Databricks account executives to get the help . <BR /><STRONG>2. WorkspaceURL</STRONG> “<SPAN><A href="https://adb-XXXXXXXXXXXX.XX.azuredatabricks.net/" target="_blank" rel="noopener">https://adb-XXXXXXXXXXXX.XX.azuredatabricks.net/</A></SPAN>"<BR /><STRONG>3. Zerobus server</STRONG> end point = “XXXXXXXXXXXX.zerobus.<REGION_NAME>.azuredatabricks.net”<BR /><STRONG>4. Target table name</STRONG> <UC.SCHEMA.TABLE> where data need to be written<BR /><STRONG>5. Service principal</STRONG> details (CLIENT_ID and CLIENT_SECRETE) find in workspace Settings > Identity and Access.</LI> </OL> <P><STRONG>Client Side : </STRONG></P> <OL> <LI>Preferred SDK (Python, Java, Rust)<BR />2. protobuff schema of table in case protobuff protocall being used.</LI> </OL> <P><STRONG>Generating Protobuf (ignore this if you are using json)</STRONG></P> <P>In Zerobus:</P> <P><EM>The .proto file defines the schema of the events written to Delta tables.<BR />Compiled client code (.py, .java, etc.) is used by producers to send data<BR />This guarantees schema correctness, performance, and compatibility at ingestion time</EM></P> <P>Protobuf defines the data contract, and the compiled files provide the language-specific code needed to efficiently send and receive that data.</P> <P>Get proto definition : </P> <P>python -m zerobus.tools.generate_proto \<BR />    --uc-endpoint "$UC_ENDPOINT" \<BR />    --client-id "$CLIENT_ID" \<BR />    --client-secret "$CLIENT_SECRET" \<BR />    --table "$TABLE_NAME" \<BR />    --output "$OUTPUT_FILE"</P> <P>this will generate XX.proto file which is Protocol Buffer definition. In next step we can use it to <STRONG>compile</STRONG> into language-specific proto </P> <P>python -m grpc_tools.protoc --python_out=. --proto_path=. record.proto</P> <P>this will generate python complied proto definition which can be used by python SDK to serialised the message.</P> <P><STRONG>Sending messages using SDK </STRONG></P> <P>load config : (give information to client about zero bus server)</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="anujsen18_0-1766502911635.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22490i753DE86A9A1BE05D/image-size/large?v=v2&px=999" role="button" title="anujsen18_0-1766502911635.png" alt="anujsen18_0-1766502911635.png" /></span></P> <P> </P> <P>Defining Configuration for Zerobus server</P> <P>Create Stream : (open stream with zerobus server using config)</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="anujsen18_1-1766502911638.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22489i9991658E11EBA679/image-size/large?v=v2&px=999" role="button" title="anujsen18_1-1766502911638.png" alt="anujsen18_1-1766502911638.png" /></span></P> <P> </P> <P>opening zerobus stream </P> <P>Send Records to Server  : (Async or Sync )</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="anujsen18_2-1766502911642.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22488i1CBB68772AEDB7C8/image-size/large?v=v2&px=999" role="button" title="anujsen18_2-1766502911642.png" alt="anujsen18_2-1766502911642.png" /></span></P> <P> </P> <P>Send messages to Zerobus</P> <P>The Final wrapper : (calling all together ) </P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="anujsen18_3-1766502911645.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22491i8F2460D9FE4AA56C/image-size/large?v=v2&px=999" role="button" title="anujsen18_3-1766502911645.png" alt="anujsen18_3-1766502911645.png" /></span></P> <P> </P> <P>main runner function</P> <P>Logging from Zerobus gives clear chain how it progressed we can also see some metric </P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="anujsen18_4-1766502911649.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22492i79DF57C7DFE343AB/image-size/large?v=v2&px=999" role="button" title="anujsen18_4-1766502911649.png" alt="anujsen18_4-1766502911649.png" /></span></P> <P> </P> <P>we can see ingested data in delta table :</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="anujsen18_5-1766502911656.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22493iA7A5C5CCDBC346FC/image-size/large?v=v2&px=999" role="button" title="anujsen18_5-1766502911656.png" alt="anujsen18_5-1766502911656.png" /></span></P> <P> </P> <P>once data loaded in target table it can be processed using Spark declarative pipelines in continuous mode to process changes to downstreams.</P> <P>Example Zerobus Ingest client implementation on GitHub:-</P> <P><SPAN><A href="https://github.com/anujsen18/databricks_zerobus_client_example/tree/main" target="_blank" rel="noopener"><STRONG>GitHub - anujsen18/databricks_zerobus_client_example</STRONG><BR /><EM>Contribute to anujsen18/databricks_zerobus_client_example development by creating an account on GitHub.</EM>github.com</A></SPAN></P> <P> </P> <H2>Limitations and Pricing Note</H2> <P>Zerobus Ingest is currently in <STRONG>public preview</STRONG> and has defined throughput limits, with optimal performance when the client and endpoint run in the same region. It supports up to <STRONG>100 MB/s or ~15,000 rows per second per stream</STRONG> and provides <STRONG>at-least-once delivery guarantees</STRONG>, requiring downstream handling of potential duplicates. While Zerobus usage is <STRONG>free during preview</STRONG>, Databricks plans to introduce pricing in the future, which should be considered for production planning.</P> <H2>Conclusion:</H2> <P>Zerobus Ingest simplifies real-time data ingestion in Databricks by allowing events to be written directly into Delta tables without relying on a traditional message broker. This approach reduces <STRONG>operational complexity</STRONG>, <STRONG>minimises latency,</STRONG> and enables <STRONG>faster analytics</STRONG> for event-driven and IoT workloads.</P> <P>While Zerobus is ideal for direct event ingestion with Databricks as the primary destination, other patterns such as Kafka, Auto Loader, or CDC pipelines remain better suited for complex stream processing, file-based ingestion, or database replication. <STRONG>Selecting the right ingestion pattern depends on workload requirements, latency needs, and operational considerations.</STRONG></P> <P> </P> <P> </P> <P> </P> Wed, 28 Jan 2026 13:53:24 GMT Anuj_Diggibyte 2026-01-28T13:53:24Z [PARTNER BLOG] Zerobus Ingest on Databricks https://community.databricks.com/t5/technical-blog/partner-blog-zerobus-ingest-on-databricks/ba-p/142433 <P>Simplifying Real-Time Streaming Data Ingestion on Databricks</P> Wed, 28 Jan 2026 13:53:24 GMT https://community.databricks.com/t5/technical-blog/partner-blog-zerobus-ingest-on-databricks/ba-p/142433 Anuj_Diggibyte 2026-01-28T13:53:24Z