Zerobus Ingest in Action: How to Stream Event Data Directly into Your Lakehouse

Giselle_Go_DB
Databricks Employee
Databricks Employee
‎10-30-2025 08:04 AM

With Zerobus Ingest, part of Lakeflow Connect. now in Public Preview, let’s take a closer look and recap what Zerobus Ingest is and share new hands-on examples across different ingestion patterns: from telemetry streaming in Databricks Apps to custom REST endpoints and message queue forwarding.

What Is Zerobus Ingest?

Zerobus Ingest is a managed service that streams event data directly into Delta tables, without staging layers, message buses, or complex orchestration. Designed for high-throughput, near real-time delivery, it simplifies architecture, accelerates time to insight, and scales for workloads like IoT telemetry, clickstream analytics, and operational event feeds.

With lightweight APIs, SDKs, and OpenTelemetry support, you can push data directly to Databricks with minimal setup. Integrated with Unity Catalog for access control, data lineage, and auditability, Zerobus ensures that every write is governed, secure, and analytics-ready within the Databricks Data Intelligence Platform.

Zerobus Ingest allows data producers to bypass the message bus and push events directly into managed Delta tables in your lakehouse.Zerobus Ingest allows data producers to bypass the message bus and push events directly into managed Delta tables in your lakehouse.
Zerobus Ingest allows data producers to bypass the message bus and push events directly into managed Delta tables in your lakehouse.

Why It Matters for Practitioners

Zerobus Ingest removes three key bottlenecks common in event streaming architectures and provides:

  • Simplified architecture: Eliminate Kafka-style data hops and reduce infrastructure overhead.
  • Reduced latency: Push data directly from producers to Delta tables in near real time.
  • Unified governance: All writes are governed through Unity Catalog, providing fine-grained access control, data lineage, and auditability.

The result: a faster, simpler, and more secure ingestion path that scales easily and reduces operational overhead.

Zerobus Ingest supports a flexible set of interfaces to fit your architecture—SDKs for Python, Java, and Rust, and REST or gRPC APIs for streaming data from any source. These options let practitioners connect applications, devices, and pipelines directly to the Databricks Lakehouse Platform, without adding new infrastructure or complexity.

Put Zerobus Ingest to Work with Hands-On Tutorials

In addition to the documentation, Databricks has published three tutorials demonstrating how Zerobus Ingest supports diverse ingestion patterns:

  • Databricks Direct Write App: Use the Zerobus Ingest Direct Write API for high-speed, multi-writer telemetry ingestion from Databricks Apps.
  • Zerobus Ingest Station: Build custom FastAPI endpoints with validation and transformation logic.
  • Zerobus Ingest RabbitMQ Forwarder: Deploy lightweight agents that stream on-premise or edge data directly to Delta tables.

Databricks Direct Write App: Zerobus Ingest Focus

Use case: App-based telemetry and structured data ingestion directly into Delta tables.

The Direct Write App demonstrates how to stream structured data from a Databricks App using the Zerobus Direct Write API for high-throughput, low-latency ingestion.
It leverages FastAPI, OAuth-based authentication, and Protobuf serialization to send data directly into Unity Catalog–managed Delta tables, all within a simple web interface.

  • Compare ingestion methods side-by-side (Direct Write vs. JDBC).
  • View real-time ingestion metrics directly in the app.
  • Deploy seamlessly using Databricks Apps and Service Principals.

Read the blog: Databricks Direct Write App

Zerobus Ingest Station: Streamlining Data Ingestion into Databricks

Use case: Partner and API-based ingestion via custom REST endpoints.

Zerobus Ingest Station is a FastAPI-based container service that wraps the Zerobus Ingest SDK to create customizable ingestion endpoints. This approach lets you define validation logic and transformations before pushing records to Zerobus Ingest, ideal for external partners, IoT devices, or REST integrations.

  • Build REST endpoints tailored to your domain schema.
  • Deploy on Databricks Apps or as a containerized service.
  • Securely route incoming data directly into Delta tables.

Read the blog: Zerobus Ingest Station

Zerobus Ingest RabbitMQ Forwarder: Streamline Data Ingestion

Use case: On-premise or message queue ingestion into the Lakehouse.

The RabbitMQ Forwarder demonstrates how to create lightweight agents that subscribe to RabbitMQ topics and forward events directly into Delta tables via Zerobus Ingest.

This is especially useful in manufacturing or industrial IoT settings where data originates from isolated or constrained networks.

  • Deploy minimal Python-based forwarding agents.
  • Connect message brokers directly to Delta tables.
  • Unify on-prem and cloud ingestion through Zerobus Ingest APIs.

Read the blog: Zerobus Ingest RabbitMQ Forwarder

Learn More and Get Started

Zerobus Ingest is currently available in Public Preview. To enable Zerobus Ingest in your workspace and start building, contact your Databricks account representative.

Explore the full set of resources below:

Zerobus Ingest represents a new approach to event data ingestion: direct, governed, and adaptable. Whether you’re streaming telemetry, partner data, or IoT events, you can now go from source to Delta table in seconds without the complexity of traditional messaging infrastructure.

By integrating directly with Lakeflow Connect and Unity Catalog, Zerobus Ingest transforms ingestion into a first-class, serverless capability within the Databricks Data Intelligence Platform.