The Lakeflow Spark Declarative Pipelines (SDP) Kafka sink is now generally available, turning your declarative pipeline into a unified engine for both data ingestion and real-time egress. You can now publish curated, governed data directly to Apache Kafka and Azure Event Hubs, enabling your operational systems to react the instant data is ready. This enables a continuous flow: scoring transactions for real-time fraud and risk detection, tightening streaming SLAs by replacing batch hand-offs with immediate delivery, and powering reverse-ETL activation into CRMs and microservices. The same pipeline that transforms your data now delivers it, with delivery monitored and governed alongside your transformations.
1. What general availability provides
GA means the sink API is committed and supported for production workloads. Concretely, GA brings:
- NEW! Production support and stability guarantees. The API surface is committed and no longer subject to preview-stage changes.
- NEW! Unity Catalog governed credentials. Authentication uses UC service credentials instead of JAAS secrets spread across pipelines, so access is centralized, auditable, and consistent across AWS MSK, Azure Event Hubs, and Google Managed Kafka.
- NEW! Multi-sink fan-out. A single pipeline can declare multiple sinks and publish to several destinations at once using multiple flows.
- NEW! Append and update flows. append_flow handles streaming inserts, and update_flow emits only changed rows for change delivery.
- Managed reliability. Lakeflow SDP handles checkpointing, retries, and recovery for the sink, and the mature Structured Streaming Kafka connector exposes its full option set, including failOnDataLoss, partitioning, and idempotent-write controls.
- Built-in observability. Delivery is monitored through the Lakeflow pipeline UI and event log, alongside your broker's own dashboards.
- Governance controls. Serverless egress control keeps network egress to your Kafka destinations under policy.
2. Why stream data to Kafka from your pipeline?
SDP sinks enable a range of operational use cases:
- Real-time fraud and risk scoring. A pipeline consumes transaction or deposit events, scores them, and emits flagged events to a Kafka topic that an operational system acts on in real time.
- Tightening streaming SLAs. A streaming workload emits results to the sink as each record is processed, replacing a batch hand-off with continuous delivery and cutting end-to-end latency from tens of seconds to seconds or below.
- Reverse ETL and activation. Publishing curated, governed lakehouse data back to operational systems such as microservices, CRMs, personalization engines, and other non-Databricks applications, without a bespoke bridge job.
- Event-driven workflows and anomaly detection. Triggering downstream processes the instant an event clears your quality rules, and streaming telemetry or sensor features out to alerting systems.
3. Customer spotlight: Clinician Nexus and the "Red Stapler" architecture
At Data + AI Summit 2025, Clinician Nexus presented Red Stapler, a system built to combine two streams of healthcare data, files ingested in bulk and edits made by users in real time, into one governed, up-to-date view that downstream systems could consume immediately. A single Lakeflow Declarative Pipeline merges both streams, applies schema and data-quality governance, and stores every record, valid or not, in an SCD Type 2 table that captures full history and gives immediate quarantine views of invalid data. The pipeline runs on Lakeflow Declarative Pipelines Serverless, publishing governed events to a Kafka-compatible bus for downstream consumers.
Red Stapler shows the core pattern of this post in production: a declarative pipeline paired with a Kafka-compatible bus so that governed events reach every downstream consumer the moment they are produced.
"Our goal was to make the path from curated data to downstream action simple, one system instead of a chain of moving parts. Lakeflow Declarative Pipelines got us there: the same pipeline that validates our human-curated data now publishes it straight to Kafka through the sink, with no separate bridge job to build or operate. Any downstream application can subscribe to governed events the moment they're produced, and our schema and quality contracts are enforced end to end. What used to be several stitched-together jobs is now a single declarative pipeline." Dwight Whitlock, Clinician Nexus
Watch the talk (Data + AI Summit 2025)
4. How it works: create_sink + a flow
Setting up a Kafka sink takes two steps: declare the sink with create_sink(name, format, options), then feed it with a flow. The two flow types map directly to Structured Streaming output modes: append_flow uses append mode, writing each row to the sink that do not change in future triggers, while update_flow uses update mode, emit all the rows that changed in a micro-batch (for example, the current total for a key in a running aggregation).
from pyspark import pipelines as dp
from pyspark.sql.functions import to_json, struct
# 1. Declare the Kafka sink
dp.create_sink(
name="my_kafka_sink",
format="kafka",
options={
"kafka.bootstrap.servers": "host:port",
"topic": "my_topic",
# ... auth options (see the multi-cloud section)
},
)
# 2. Stream curated data into it
@dp.append_flow(name="to_kafka", target="my_kafka_sink")
def to_kafka():
df = dp.read_stream("silver_table")
return df.select(to_json(struct("*")).alias("value"))
Note : All Structured Streaming Kafka options are supported, since this uses the same connector rather than a new implementation.
5. Wherever your Kafka runs, one API
Teams run Kafka anywhere, on Confluent Cloud, Amazon MSK, Azure Event Hubs, or Google Cloud Managed Kafka, and the same create_sink(..., format="kafka", ...) call targets all of them; only the bootstrap servers and authentication change (Connect to Kafka).
5a. Confluent Cloud (runs on any of the three clouds), SASL_SSL
The most portable option: an API key and secret stored in a Databricks secret scope.
api_key = dbutils.secrets.get(scope="kafka-sink", key="confluentApiKey")
api_secret = dbutils.secrets.get(scope="kafka-sink", key="confluentSecret")
JAAS_CONFIG = (
"kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required "
f"username='{api_key}' password='{api_secret}' ;"
)
dp.create_sink(
name="my_kafka_sink",
format="kafka",
options={
"kafka.bootstrap.servers": "pkc-xxxxx.us-west-2.aws.confluent.cloud:9092",
"topic": "cookie_topic",
"kafka.security.protocol": "SASL_SSL",
"kafka.sasl.mechanism": "PLAIN",
"kafka.sasl.jaas.config": JAAS_CONFIG,
"kafka.ssl.endpoint.identification.algorithm": "https",
"failOnDataLoss": "false",
},
)
5b. Azure Event Hubs (Kafka endpoint) with a Unity Catalog service credential
The GA-recommended, secret-less pattern, with credentials governed in Unity Catalog:
credential_name = "<service-credential>"
eh_namespace_name = "dp-eventhub"
dp.create_sink(
name="eh_sink",
format="kafka",
options={
"databricks.serviceCredential": credential_name,
"kafka.bootstrap.servers": f"{eh_namespace_name}.servicebus.windows.net:9093",
"topic": "dp-sink",
},
)
5c. AWS (Amazon MSK) and GCP (Google Managed Service for Apache Kafka)
Both use the same databricks.serviceCredential pattern; just point kafka.bootstrap.servers at your MSK or Google Managed Kafka brokers:
dp.create_sink(
name="my_kafka_sink",
format="kafka",
options={
"databricks.serviceCredential": "<service-credential>",
"kafka.bootstrap.servers": "<msk-or-google-bootstrap>:9092",
"topic": "cookie_topic",
},
)
6. Real-time mode: when milliseconds matter
For operational use cases that need ultra-low latency, Lakeflow SDP integrates with real-time mode, which delivers end-to-end latency as low as five milliseconds (Use real-time mode in Lakeflow SDP). Real-time mode is a specialized continuous trigger that adds three latency optimizations on top of continuous mode: long-running batches, simultaneous stage scheduling, and a streaming shuffle that passes data between stages as soon as it is produced. Apache Kafka, AWS MSK, and Azure Event Hubs are all supported as both source and sink, so a Kafka-in, transform, Kafka-out pipeline runs end to end at operational latencies. In real-time mode both source and target must be Kafka-compatible, so Delta is not supported as a real-time source or sink.
To turn on real-time mode, set two things in your pipeline settings: run the pipeline in continuous mode, and add the real-time flag to the Spark config.
{
"continuous": true,
"spark_conf": {
"spark.databricks.streaming.realTimeMode.enabled": "true"
}
}
Then define a real-time update flow with @DP.update_flow, setting pipelines.trigger to "RealTime" and targeting a Kafka sink. The example below reads a Kafka input topic and streams the transformed records straight to a Kafka output sink, all within one real-time pipeline:
from pyspark import pipelines as dp
# Output sink (create_sink API)
dp.create_sink("kafka_output_sink", "kafka", {
"kafka.bootstrap.servers": broker_address,
"topic": output_topic,
})
# Real-time update flow: Kafka source to Kafka sink
@dp.update_flow(
name="kafka_rtm_flow",
target="kafka_output_sink",
spark_conf={
"pipelines.trigger": "RealTime", # turns on real-time mode for this flow
"pipelines.trigger.interval": "5 minutes", # optional checkpoint interval; default 5 min
},
)
def kafka_rtm_flow():
return (
spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", broker_address)
.option("subscribe", input_topic)
.option("startingOffsets", "latest")
.load()
.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "timestamp")
)
With the Kafka sink in real-time mode, stateless transforms, aggregations, tumbling and sliding windows, broadcast (stream-to-static) joins, and stream-to-stream inner joins (on Databricks Runtime 18 and above, with additional Spark configuration) are supported. Stream-to-stream outer joins, foreachBatch, and session windows are not supported. See the real-time mode reference for the full list.
The low latency comes from substantial engine work in Spark Structured Streaming that cut Kafka source-to-sink latency by up to 75 percent across throughput levels (Latency goes subsecond).
7. Try it: the notebook
Validated end-to-end: create_sink(format="kafka") plus an append_flow streamed a Delta source from a serverless Lakeflow pipeline to Azure Event Hubs (Kafka protocol, SASL_SSL), with delivery confirmed on the Event Hubs side.
A ready-to-run pipeline notebook accompanies this post (notebooks/ldp_kafka_sink_multicloud.py). It reads the bakehouse.sales.transactions dataset (free on Databricks Marketplace), applies a @DP.expect_or_drop data-quality expectation and a business filter, serializes high-value purchases to JSON, and streams them to a Kafka topic, with ready-to-uncomment auth blocks for Confluent Cloud, Amazon MSK, Azure Event Hubs, and Google Managed Kafka. There is also an official, maintained demo in the Databricks tech-marketing repo: databricks/tmm Lakeflow-SDP-Kafka-Sink.
Quick start (Confluent example):
databricks secrets create-scope kafka-sink
databricks secrets put-secret kafka-sink confluentApiKey --string-value <api-key>
databricks secrets put-secret kafka-sink confluentSecret --string-value <secret>
Then point a Lakeflow pipeline at the notebook, set your BOOTSTRAP server and TOPIC, and run it. Watch records land on your topic in the Lakeflow pipeline UI and your broker's console.
8. Get started
Docs: Using sinks in pipelines · create_sink reference · Connect to Kafka
Release notes: May 2026 platform release notes
Real-time mode: Structured Streaming real-time mode
Demo repo: databricks/tmm
The lakehouse can now both ingest and emit streaming data through a single declarative framework. For teams currently bridging Lakeflow to a message bus with a separate Structured Streaming job, the Kafka sink consolidates that path into the pipeline itself.
