https://community.databricks.com/t5/technical-blog/lakeflow-connect-message-bus-ingestion-now-shipping-your-logs/ba-p/167458 <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Vicky_Bukta_DB_1-1788469144356.jpeg" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/30686i3452E443E10689DB/image-size/large?v=v2&amp;px=999" role="button" title="Vicky_Bukta_DB_1-1788469144356.jpeg" alt="Vicky_Bukta_DB_1-1788469144356.jpeg" /></span></P> <P><SPAN>Take a quick inventory of where your operational data lives right now. There is probably a </SPAN><STRONG>Kafka</STRONG><SPAN> cluster carrying your event backbone, or perhaps </SPAN><STRONG>RabbitMQ</STRONG><SPAN> capturing messages on a factory floor or between services. Getting this data into your lakehouse for analytics has traditionally meant standing up, scaling, and babysitting a custom ingestion pipeline for every topic or queue.</SPAN></P> <P><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/streaming-overview" target="_blank" rel="noopener"><STRONG>Lakeflow Connect</STRONG></A><SPAN> now eliminates this maintenance burden. </SPAN><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/kafka" target="_blank" rel="noopener"><STRONG>Kafka</STRONG></A><STRONG> and </STRONG><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/rabbitmq" target="_blank" rel="noopener"><STRONG>RabbitMQ</STRONG></A><SPAN> ingestion are now available as </SPAN><STRONG>fully managed streaming connectors (Beta)</STRONG><SPAN>,&nbsp; allowing you to bring your most demanding operational streams into the lakehouse with zero infrastructure to manage. You simply point a connector at your source stream, and the data flows continuously into Delta.</SPAN></P> <H2><SPAN>Your workload changes, your pipelines should too</SPAN></H2> <P><SPAN>Lakeflow Connect for message buses is packed with features to handle your toughest workloads. Advanced functionality, such as bin-packing under the hood, allows us to effectively distribute and bundle your topic ingestion across compute resources for optimal performance.&nbsp;</SPAN></P> <P><SPAN>Think back to how you managed your low- to moderate-workload. How do they change and fluctuate over the course of a week or month? It's common to see events spike during peak periods, such as flash sales, working hours, or newsletters being sent out. This translates to fluctuations in volume for your topics. Having separate dedicated pipelines is great for durability, but can be costly. On the other hand, bundling your topics into a single pipeline can be messy to maintain when scaling clusters is required for matriculation.</SPAN></P> <P><SPAN>Bin-packing allows managed Lakeflow Connect streaming ingestion to use a single CPU core to process data from multiple Kafka partitions within a single task more efficiently. Lakeflow Connect is also serverless, which means that you only pay for the compute you use, rather than the compute you provisioned. The result is a cost-effective pipeline that can scale elasticity in response to your changing volumes.</SPAN></P> <H2><SPAN>Getting started with the managed connectors</SPAN></H2> <P><SPAN>These managed connectors are configured today with </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/bundles/" target="_blank" rel="noopener"><STRONG>Declarative Automation Bundles (DABs)</STRONG></A><STRONG>, </STRONG><SPAN>with the exception of RabbitMQ, which is coming soon, and the pipeline API, which means you can author them from a notebook or check them into your source code. Point-and-click </SPAN><STRONG>UI authoring is coming</STRONG><SPAN>.</SPAN></P> <P><SPAN>Each connector uses two pieces:</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><SPAN>A </SPAN><STRONG>Connection</STRONG><SPAN>, a Unity Catalog securable that stores the broker endpoint and credentials. You create it once in </SPAN><STRONG>Catalog Explorer</STRONG><SPAN> (choose the connector type and enter the bootstrap servers or broker endpoint plus authentication), and reference it by name in your pipelines.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>An </SPAN><STRONG>ingestion pipeline</STRONG><SPAN>, defined as a bundle, that maps sources to destination tables and runs on </SPAN><STRONG>serverless compute</STRONG><SPAN>.</SPAN></LI> </OL> <H3><SPAN>Example</SPAN></H3> <P><SPAN>Here is a minimal Kafka ingestion pipeline. It reads two topics into a single destination table.</SPAN></P> <P><SPAN>This example demonstrates the following:</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Selecting the UC connection to our Kafka cluster via </SPAN><SPAN>connection_name</SPAN><SPAN>.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Describing the managed Kafka pipeline via </SPAN><SPAN>kafka_pipeline</SPAN></LI> <OL> <LI style="font-weight: 400;" aria-level="2"><SPAN>Choosing a destination via destination_catalog, destination_schema, which represents the schema, and destination_table. Note that </SPAN><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/kafka-reference#fanout-options" target="_blank" rel="noopener"><SPAN>fanout rules</SPAN></A><SPAN> can be specified to autocreate tables.</SPAN></LI> </OL> </OL> <LI-CODE lang="markup">variables: connection_name: default: my-kafka-connection dest_catalog: default: main dest_schema: default: kafka_ingest resources: pipelines: kafka_pipeline: name: kafka-ingestion-pipeline continuous: true channel: PREVIEW catalog: ${var.dest_catalog} target: ${var.dest_schema} ingestion_definition: connection_name: ${var.connection_name} objects: - table: source_table: N/A destination_catalog: ${var.dest_catalog} destination_schema: ${var.dest_schema} destination_table: user_events connector_options: kafka_options: topics: [user-events, power-user-events]</LI-CODE> <P><SPAN>Deploy it with the bundle CLI:</SPAN></P> <LI-CODE lang="python">databricks bundle init databricks bundle deploy</LI-CODE> <P><SPAN>The connector handles the rest: reading continuously from the topics and writing to the </SPAN><SPAN>user_events</SPAN><SPAN> streaming table.</SPAN></P> <P><STRONG>Flexible data handling with Variant</STRONG></P> <P><SPAN>&nbsp;Kafka messages are rarely perfectly uniform, so the connector lets you say how to deserialize values and what to do when the shape drifts. This is where it connects to schema management directly.</SPAN></P> <P><SPAN>This example demonstrates the following:</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><SPAN>format</SPAN><SPAN> is set to </SPAN><SPAN>STRING</SPAN><SPAN> and </SPAN><SPAN>JSON</SPAN><SPAN>. The </SPAN><SPAN>JSON</SPAN><SPAN> body of the message will then be parsed to a variant column for ultimate flexibility.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Starting_offset is set to the latest signal from which we should start ingestion in the topic. Options are the <FONT face="andale mono,times">latest</FONT> and the </SPAN><FONT face="andale mono,times">earliest</FONT><SPAN>.</SPAN><SPAN>&nbsp;Learn more from our documentation </SPAN><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/kafka-reference#transformer-options" target="_blank" rel="noopener"><SPAN>here</SPAN></A><SPAN>.</SPAN></LI> </OL> <LI-CODE lang="markup">ingestion_definition: connection_name: ${var.connection_name} objects: - table: source_table: N/A destination_catalog: ${var.dest_catalog} destination_schema: ${var.dest_schema} destination_table: user_events connector_options: kafka_options: topics: - user-events starting_offset: latest key_transformer: format: STRING value_transformer: format: JSON</LI-CODE> <P><STRONG>Schema Inference and Evolution.</STRONG></P> <P><SPAN>Though schemas drift slowly, teams typically establish rules for schema evolution to ensure backward compatibility (e.g., adding optional columns or widening types). Lakeflow Connect automatically applies forward-rolling schema updates, maintaining schematized data and optimal query performance</SPAN></P> <LI-CODE lang="markup">ingestion_definition: connection_name: ${var.connection_name} objects: - table: source_table: N/A destination_catalog: ${var.dest_catalog} destination_schema: ${var.dest_schema} destination_table: user_events connector_options: kafka_options: topics: - user-events starting_offset: latest key_transformer: format: STRING value_transformer: format: JSON json_options: schema_evolution_mode: ADD_NEW_COLUMNS</LI-CODE> <P><STRONG>Routing to many tables (fanout).</STRONG><SPAN>&nbsp;</SPAN></P> <P><SPAN>Other times, it's not about schema drift. It's the fact that a single Kafka topic could contain many different event types. Our fanout capability lets you route events to different tables using an expression, rather than defining a pipeline per table.</SPAN></P> <LI-CODE lang="markup">objects: - schema: source_schema: N/A destination_catalog: ${var.dest_catalog} destination_schema: ${var.dest_schema} connector_options: kafka_options: topic_pattern: 'events-.*' starting_offset: earliest value_transformer: format: STRING fanout_options: fanout_by: 'cast(value as string):event_type::string' transforms: - format: JSON input_column: value</LI-CODE> <H2><SPAN>Decision matrix: managed connector or forwarder?</SPAN></H2> <P><SPAN>There are two approaches to move your data: Push vs. pull.</SPAN></P> <P><SPAN>Our </SPAN><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/streaming-overview" target="_blank" rel="noopener"><STRONG>managed streaming connectors</STRONG></A><SPAN> are </SPAN><STRONG>pull-based</STRONG><SPAN>. Lakeflow Connect reaches into your source, polls for data, and replicates it. Due to the fact that it reaches in, the connector </SPAN><STRONG>requires inbound network access</STRONG><SPAN> to the source. In exchange, there is </SPAN><STRONG>no software for you to run or maintain</STRONG><SPAN>. It is a fully managed service.</SPAN></P> <P><SPAN>A </SPAN><A href="https://docs.databricks.com/aws/en/ingestion/zerobus-overview" target="_blank" rel="noopener"><STRONG>Zerobus Ingest forwarder</STRONG></A><SPAN> is </SPAN><STRONG>push-based</STRONG><SPAN>. A sidecar process you build with the Zerobus SDK runs inside your perimeter and pushes data out, so all traffic is </SPAN><STRONG>outbound only</STRONG><SPAN>. Zerobus is flexible enough that you can build a forwarder from almost </SPAN><STRONG>any source system</STRONG><SPAN>. The cost is that you own and run that agent.&nbsp;</SPAN></P> <P><SPAN>Our connector and Zerobus Ingest services are therefore complementary, rather than competing.&nbsp;&nbsp;</SPAN><SPAN>Evaluate the tradeoff between the managed connector and the Zerobus forwarder based on your security posture and operational constraints, as outlined in the matrix below.<BR /><BR /></SPAN></P> <TABLE> <THEAD> <TR> <TH>&nbsp;</TH> <TH> <P><STRONG>Managed streaming connector</STRONG></P> </TH> <TH> <P><STRONG>Zerobus forwarder</STRONG></P> </TH> </TR> </THEAD> <TBODY> <TR> <TD> <P><STRONG>Direction</STRONG></P> </TD> <TD> <P><STRONG>Pull.</STRONG><SPAN> Lakeflow Connect replicates from the source</SPAN></P> </TD> <TD> <P><STRONG>Push.</STRONG><SPAN> Your agent pushes records out</SPAN></P> </TD> </TR> <TR> <TD> <P><STRONG>Who runs it</STRONG></P> </TD> <TD> <P><SPAN>Fully managed by </SPAN><STRONG>Lakeflow Connect</STRONG><SPAN>. No software to run</SPAN></P> </TD> <TD> <P><STRONG>You build and run </STRONG><SPAN>the agent (using the Zerobus SDK)</SPAN></P> </TD> </TR> <TR> <TD> <P><STRONG>Network</STRONG></P> </TD> <TD> <P><SPAN>Requires </SPAN><STRONG>inbound</STRONG><SPAN> access to the source</SPAN></P> </TD> <TD> <P><STRONG>Outbound only</STRONG><SPAN> from inside your perimeter</SPAN></P> </TD> </TR> <TR> <TD> <P><STRONG>Source of truth</STRONG></P> </TD> <TD> <P><SPAN>Stays in the </SPAN><STRONG>message bus</STRONG></P> </TD> <TD> <P><SPAN>Is the </SPAN><STRONG>lakehouse.</STRONG></P> </TD> </TR> <TR> <TD> <P><STRONG>Sources</STRONG></P> </TD> <TD> <P><SPAN>Kafka, RabbitMQ, other coming soon.</SPAN></P> </TD> <TD> <P><STRONG>Any source</STRONG><SPAN> you can write code against.</SPAN></P> </TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Vicky_Bukta_DB_0-1788469122486.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/30685iE423AFDD6CF37BD7/image-size/large?v=v2&amp;px=999" role="button" title="Vicky_Bukta_DB_0-1788469122486.png" alt="Vicky_Bukta_DB_0-1788469122486.png" /></span></P> <P>&nbsp;</P> <H2><SPAN>What's next?</SPAN></H2> <P><SPAN>The managed streaming connectors are just getting started. UI-based authoring is on the way and more data sources are coming soon. On the horizon, we are adding native support for </SPAN><STRONG>Avro</STRONG><SPAN> and </SPAN><STRONG>Protobuf</STRONG><SPAN>, including full </SPAN><STRONG>schema registry integration</STRONG><SPAN>. Full UI experience is around the corner.</SPAN></P> <P><SPAN>Zerobus Ingest is adding </SPAN><A href="https://docs.databricks.com/aws/en/ingestion/zerobus-kafka" target="_blank" rel="noopener"><STRONG>Apache Kafka-compatible</STRONG></A><SPAN> and </SPAN><STRONG>MQTT-compatible</STRONG><SPAN> producer APIs for the push-based side of the story. Whichever direction your data needs to move, the trade-offs in the matrix above will still guide you.</SPAN></P> <P><SPAN>Ready to try it? Check out the </SPAN><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/kafka" target="_blank" rel="noopener"><SPAN>Kafka </SPAN></A><SPAN>and </SPAN><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/rabbitmq" target="_blank" rel="noopener"><SPAN>RabbitMQ connector docs</SPAN></A><SPAN> to set up your first managed pipeline.</SPAN></P> <P><I><SPAN>Have questions, or want to share which message buses you are bringing into your lakehouse? Join the discussion below!</SPAN></I></P> Fri, 04 Sep 2026 17:21:25 GMT Vicky_Bukta_DB 2026-09-04T17:21:25Z https://community.databricks.com/t5/technical-blog/lakeflow-connect-message-bus-ingestion-now-shipping-your-logs/ba-p/167458 <DIV class="kix-appview-editor-container"> <DIV class="kix-appview-editor"> <DIV data-grammarly-part="highlights"> <DIV class="AXzy4" data-grammarly-part="highlight-group"> <DIV class="gUhBO jlbiw oR786 fX3Dk" data-highlight-color-name="blue" data-highlight-format="enclosing" data-grammarly-part="highlight" data-key="1000274-323|0-1"> <DIV class="wPV13"> <P>Effortlessly ingest Kafka and RabbitMQ data into your Lakehouse with Lakeflow Connect's new fully managed streaming connectors. Scale without the maintenance.</P> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> Fri, 04 Sep 2026 17:21:25 GMT https://community.databricks.com/t5/technical-blog/lakeflow-connect-message-bus-ingestion-now-shipping-your-logs/ba-p/167458 Vicky_Bukta_DB 2026-09-04T17:21:25Z