Disclosure: I work at Onibex, the company that builds OneConnect, which is used as the worked example in this post. This is about the engineering approach, not a product pitch.
Design of Databricks Pipelines for real-time data transformation to generate the Silver and Gold tables of the medallion architecture
A sales order in SAP doesn't live in one place. It's born split between a header (VBAK) and its line items (VBAP). This post walks through how that order can be made available in Databricks, updated at the moment it changes, without anyone having to touch ABAP or hand-build a Kafka pipeline.
This post walks through that path end to end: from modeling VBAK and VBAP inside SAP, to a live Kafka topic, to a Delta table ready to build a Databricks pipeline on top of. I use the sales order as the example, because it's a good case of SAP's habit of splitting one business concept across tables that were never meant to be read together.
Modeling the Entity in SAP
Modeling the sales order goes through a low-code Data Modeler that runs inside SAP. You bring in VBAK (the header: customer, document date, sales organization) and VBAP (the line items: material, quantity, net value), connect them visually by dragging and dropping, and expose the fields you need under business aliases. NETWR becomes "net value." KUNNR becomes "customer." All of this without writing SQL, without ABAP function modules, without depending on a developer for every change.
The result of that work isn't just another table. It's a data product: a complete business package ("sales order"), with its tables, relationships, and aliases already resolved. Someone builds it once, and from there it's available for any future project that needs it.
Data product design for Sales Order in the Data Modeler
No special configuration is needed for this to land cleanly in Databricks afterward. You model the sales order the same way you'd model it for any other destination, Snowflake, PostgreSQL, a dashboard. The data comes out already structured; Databricks doesn't need anything extra on the SAP side.
The Data Modeler also supports custom Z-tables and CDS Views, not just standard tables. If your SAP installation has years of custom fields added to the sales order, or if your functional team already built a clean CDS View for internal reporting, it can be pulled in directly, without rebuilding that logic from scratch.
The Business Event: Real-Time Notification
Modeling the tables is only half the work. The other half is knowing the moment something changed, without having to constantly ask SAP whether something new happened.
Every time a sales order is created, modified, or deleted in SAP, a business event tied to that entity fires. By standard, this runs on BOR Events, and on S/4HANA 2023 installations onward, on RAP Events. Whichever framework it is, the result is the same: the modified record travels immediately toward the rest of the ecosystem, without waiting for a scheduled job to check whether something changed.
This is what guarantees that the information later used by Databricks, a dashboard, or an AI agent, is never out of date.
Example of a Power BI dashboard connected to Databricks, using the VBAK, VBAP, VBRK, VBRP tables, etc.
Turning the Event Into a Kafka Topic
Once VBAK and VBAP are modeled and connected to their business events, a gateway component takes over. It does four things, all automatic:
First, it receives the delta from SAP: the modified record, whether an insert or an update, at the moment the business event fires.
Second, it serializes each record into Avro format instead of raw JSON. This matters more than it seems: Avro forces every field to have a defined type and validates every record before writing it, instead of letting a field be a string in one record and null in the next, which is exactly the kind of inconsistency that breaks a downstream pipeline without warning.
VBAK kafka topic message serialized in Avro
Third, it registers and manages the schema for each data product in the Schema Registry, a centralized repository of every schema the Kafka topics use. Nobody wrote that schema by hand. It was generated the moment the data product was activated, from the same metadata already known about VBAK and VBAP.
Avro schema registry for the VBAK kafka topic
Fourth, it creates the Kafka topic for the data product automatically, with a name consistently based on the entity, all of this running on Apache Kafka. From that point on, every change to the sales order writes a new record to the topic, in Avro format, with the schema enforced by the registry.
From Kafka Topic to Delta Table
This is where a Kafka Connect connector for Databricks comes in, configured directly from the gateway instead of needing a separate Kafka Connect deployment operated on its own.
Pre-loaded connectors in the Smart Gateway
That connector reads the sales order topic using the same schema registry entry the gateway wrote. The input schema and the output schema are guaranteed to match, because they're literally the same contract read on both ends, with no separate mapping step in the middle where something could drift out of alignment.
Easy configuration to create connectors
What lands in Databricks isn't a raw copy. Each SAP table (VBAK, VBAP) arrives as its own Delta table, continuously synced, with enableChangeDataFeed turned on by default. That means Delta records every insert, update, and delete into a feed that can be read as a stream downstream, instead of only showing the current state of the table. That property is exactly what makes it possible to build a real CDC pipeline on top of it, instead of having to reprocess full snapshots on every run.
Automatic configuration of the tables created with the Databricks connector, including the enableChangeDataFeed property
From Bronze to Gold, With the Sales Order as the Example
With VBAK and VBAP landing as bronze tables with change feed enabled, the pattern for building the silver and gold layers is a familiar one: read the change feed as a stream, apply the changes onto a "current" table using SAP's real business keys, and propagate deletes with apply_as_deletes instead of leaving ghost records behind.
For the sales order, that means a consolidated silver that joins VBAK and VBAP into a single reusable entity, with column names that preserve traceability back to the source table. From there, the gold layer enriches with the text tables (customer name, material description), derives the calculated fields the business needs, and renames everything to business terms: what was vbeln, matnr, and netwr in bronze becomes sales_order, material, and net_value in gold.
That's the kind of table a human, a dashboard, or an AI agent can read without an SAP dictionary sitting next to them.
Design of Databricks Pipelines for real-time data transformation to generate the Silver and Gold tables of the medallion architecture
Why This Combination Matters
Each piece in this chain solves something the others can't. The Data Modeler removes the need for an ABAP developer every time you want to model a new SAP entity, including custom Z-tables and CDS Views. The business event removes the need to poll SAP on a fixed schedule, because the system itself notifies the change the instant it happens. The gateway removes the need to build Kafka infrastructure, schemas, and topics by hand, and makes schema management fully automatic. The Databricks connector removes the need for a custom Kafka Connect deployment and guarantees the schema stays consistent from SAP all the way to Delta. And enableChangeDataFeed is what makes the CDC pipeline possible from the very first moment, instead of forcing full snapshot reprocessing on every run.
Take any one of these pieces away and you're back to something manual: mapping schemas by hand, polling SAP on a fixed schedule, reconciling a batch load against yesterday's numbers, or having a pipeline that silently breaks every time someone adds a field in SAP and nobody says anything.
What This Unlocks
Before this kind of architecture, having the sales order available in Databricks meant, at best, an overnight job that loaded a snapshot and hoped nobody needed a number fresher than the previous night's. At worst, it meant an ABAP developer building a custom extraction that worked fine until that person left the company or SAP released a patch that broke the logic without warning.
Today, a sales order created in SAP this morning is a queryable gold table in Databricks by the time anyone asks about it, no overnight job, no reconciliation step. That's the foundation for a real-time revenue dashboard, a forecasting model that needs order data updated to the minute, or an AI agent answering "how much does customer X have on order this week" without anyone having to translate VBAK.NETWR into a sentence a business user can read.
None of this required a single line of ABAP, and none of it required standing up Kafka Connect infrastructure by hand. The whole chain, from the Data Modeler to the gateway to a native Databricks connector, is built to hand off cleanly from one layer to the next, without the work of one stage ever needing to be redone at the next.
Frequently Asked Questions
Do I need to write ABAP code to get SAP data into Databricks with this flow?
No. The Data Modeler exposes a visual drag-and-drop interface for defining the data product, and the gateway automatically manages schemas, topics, and serialization into Kafka.
What if my SAP installation has custom Z-tables?
The Data Modeler supports them the same way it supports standard tables. You insert them, configure their joins with the rest of the data product, and filter the fields that should leave SAP, all from the same interface.
How do Databricks pipelines find out something changed in SAP?
Every table arrives with enableChangeDataFeed enabled by default, which lets you read inserts, updates, and deletes as a continuous stream instead of having to compare snapshots.
Can deletes get lost in this flow?
No, because the delete in SAP is itself the business event that triggers the send. Delete propagation is part of the design from the source, not a step bolted on afterward.
Does this only work with sales orders?
The sales order is this post's example, but the same flow applies to any data product modeled this way: sales orders, billing documents, material masters, customers, vendors, and any Z-table or CDS View your SAP has configured.
AI disclosure: This post was drafted with the help of an AI writing assistant and reviewed and verified by the author before publishing.
