We just published a deep-dive engineering post on Zerobus Ingest that covers how it works and the design choices that made it possible.
1 petabyte. 24 hours. 12 GB/s sustained. 12 million rows per second. Single table. Zero pre-configuration.
The three engineering bets that helped Zerobus Ingest deliver performance at scale:
- Dynamic partitioning: Ordering is guaranteed at the stream connection level rather than the partition level, which means pods can be added and removed dynamically — true elastic autoscaling, not just scale-out.
- Zeroparser (now open sourced): A single-pass, zero-allocation protobuf decoder built in Rust that sustains ~1 GB/s per CPU core with fully dynamic descriptors — no compile-time schemas required.
- Latency-optimized write-ahead log: The WAL keeps clients lean—push data, get ack, free buffer. This low-latency, durable handoff is why teams prefer Kafka. Zerobus offers the same guarantee.
NASA's NEOWISE dataset is 11 years of Wide-field Infrared Survey Explorer data — 200 billion time-stamped detections of stars, galaxies, and asteroids with a wide, complex schema that reflects what the universe actually produces. It's actual scientific data, not a synthetic workload, and it behaves exactly like production telemetry because it is production data.
That's why we picked it.
Run the benchmark yourself
Everything we used to produce the petabyte-run is in our sample repo:
- Dataset prep
- The Locust-based producer
- Instructions for pointing it at your own Zerobus endpoint
github.com/databricks-solutions/zerobus-ingest-examples — NEOWISE benchmark
The NEOWISE run was our test of what the system can do at scale, but your pipeline is the real test. Try taking our benchmark and swapping out the data for yours 💪
What comes next?
Zerobus Ingest is Generally Available, with Kafka Producer API support, MQTT, Avro, and more on the roadmap.
Beyond what's on our list, we want to know what's on yours. Where do you want Zerobus to go next? What do you think the next frontier of streaming looks like?
This community is our best signal. Drop a comment below or reach out directly to your account team— we're actively shaping what we build next.
Get started:
- Docs: Zerobus Ingest overview
- SDK: github.com/databricks/zerobus-sdk
- Benchmark harness: NEOWISE benchmark on GitHub
