Introducing Zerobus Ingest Station: Streamlining Data Ingestion into Databricks
Welcome to your destination. The Zerobus Ingest Station!
Sometimes you just need a customized ingestion endpoint.
When you’re looking to set up an interface for partnering with other companies or connecting to public-facing from your devices, there are quite a few things to keep in mind! You’ll want to make sure it works smoothly with your current authentication system and data validation rules. It can feel a bit overwhelming with so much to juggle, but taking the time to get these elements right will set you up for success before you dive into scaling write operations for your Delta table.
Zerobus Ingest, part of Lakeflow Connect, specializes in “push-based” ingestion, streamlining ingestion to the Lakehouse by bringing you closer to the data source. It handles scalable ingestion, so that you can focus on rest.
Zerobus Ingest Station is a FAST API application which is a wrapper around the Zerobus Ingest SDK. This application provides a clear blueprint for creating custom HTTP REST endpoints, enabling you to implement specific business logic and apply transformations before pushing records to Zerobus Ingest.
curl -X POST http://localhost:8000/ingest/station_one \
-H "Content-Type: application/json" \
-d '{
"device_name": "sensor-001",
"temp": 25,
"humidity": 60
}'
You can deploy it on Databricks Apps or as a Docker container in your own cluster.
→ Check out the Github repo
Why Zerobus Ingest Station?
Traditionally, getting data into Delta Lake often required a deep understanding of Delta Lake mechanics and writing to the format directly. Zerobus Ingest Station liberates you from this complexity. By leveraging Zerobus Ingest, your custom FAST API application can focus solely on your core business requirements – defining your data, transforming it as needed, and validating its integrity. Once your data is ready, Zerobus Ingest Station handles the heavy lifting of seamlessly landing that data into your Databricks Delta tables.
This approach eliminates the need to write custom Delta Lake write logic, allowing your teams to concentrate on what matters most: getting your business logic right and delivering value faster. Zerobus Ingest Station showcases a flexible and customizable path to reliable, high-performance data ingestion into Databricks.
More in the Zerobus Ingest series: Once you’ve seen how Zerobus Ingest Station creates a customizable API layer for inbound data, check out these companion implementations to complete your ingestion toolkit:
