Today, we are introducing Beta support for Kafka-compatible producer API support in Zerobus Ingest. This is an incredible milestone for Zerobus, given the wide ecosystem built around Kafka.
Take a moment and ask yourself, “What fraction of that data is ultimately headed to your data lake?” And of that fraction, for how much of it is the lakehouse the only place it really needs to end up?
For a lot of teams, the honest answer is "most of it." The events get produced to Kafka, consumed once, and written to Delta, where the actual analytics happen. In that pattern, Kafka is not the destination. It is the pipe. Now Zerobus can be your pipe, with the added benefit that it brings you data straight to your lakehouse.
What "Apache Kafka-compatible" means here
Zerobus supports the Apache Kafka producer protocol. This means your existing Kafka producer clients and the standard libraries you already use, can seamlessly authenticate and send records to Zerobus. To be precise, Zerobus is not a Kafka broker nor is it a message bus. Zerobus implements the Apache Kafka producer API so that you can keep using the tools you already use today.
In a Kafka producer, you configure two things that matter here:
- The broker URL (bootstrap.servers). You swap it for a Zerobus endpoint.
- The topic you send to. You swap it for a fully-qualified table name, catalog.schema.table.
Change those two values, keep the client, and your records land in Delta.
Hello World: Reconfiguring your client
Let's prove the "no rewrite" claim with a plain kafka-python producer. The only Zerobus-specific piece is authentication, and it is worth understanding because it is the one part that is not just a config swap.
Zerobus authenticates with OAuth, using the standard Kafka SASL_SSL security protocol and the OAUTHBEARER SASL mechanism. Kafka clients already support that mechanism; they just need something that supplies a valid token and refreshes it. In kafka-python, that something is a token provider. Checkout our sample token handler here.
Now for the producer itself. To configure an ordinary KafkaProducer like the one below, we pass the Zerobus endpoint for the bootstrap_servers and the table name instead of topic name.
import json
from kafka.producer import KafkaProducer
# The two swaps: broker URL -> Zerobus endpoint, topic -> table name.
bootstrap_servers = "<workspace-id>.zerobus.<region>.<workspace-host>:9092"
table_name = "main.default.air_quality"
token_provider = ZerobusTokenProvider(
client_id="<client-id>",
client_secret="<client-secret>",
token_endpoint="https://<your-workspace>/oidc/v1/token",
workspace_id="<workspace-id>",
catalog="main", schema="default", table="air_quality",
)
producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol="SASL_SSL",
sasl_mechanism="OAUTHBEARER",
sasl_oauth_token_provider=token_provider,
acks="all",
)
# Send to the table as if it were a topic.
record = {"device_name": "kafka-producer", "temp": 50, "humidity": 1}
future = producer.send(
topic=table_name,
value=json.dumps(record).encode("utf-8"),
)
metadata = future.get(timeout=30)
print(f"acked: partition={metadata.partition} offset={metadata.offset}")
producer.flush()
producer.close()When a common Kafka producer. send() returns a future, you get back the familiar partition and offset metadata on acknowledgment, and acks="all" gives you the durability semantics you would expect. Nothing about the producer API changed. Only where the data goes has changed.
One thing worth calling out: the payload is JSON-encoded bytes. That lines up with the rest of Zerobus's JSON ingestion story, including schema handling and the rescue column, so the same table-contract thinking from choose-your-own-adventure schema management applies to records you send this way.
Rethinking the pipe
Come back to your inventory. Every producer on that list that exists mainly to get data into the lakehouse is a producer you can repoint. Kafka was a means, not an end, for that data, and the Apache Kafka-compatible producer APIs let you shorten the path from producer to Delta.
You do not have to move everything. Keep Kafka where it is genuinely the system of record and where multiple independent consumers depend on it. But for the data whose only real destination is the lakehouse, sending it there directly is simpler to operate and one fewer system to keep alive.
This is also just the beginning of the push-based story. Zerobus is adding MQTT-compatible producer APIs later this summer, so more of your existing producers will be able to take the same direct path.
Ready to try it? Check out the Zerobus documentation to get started. If you are pulling from an existing Kafka cluster rather than producing into Zerobus, see our managed message bus ingestion connectors.
Have questions, or want to share what you are repointing to your lakehouse? Join the discussion below!
