One way to keep your accounts receivable information current in real time on Databricks, with the change semantics written into the pipeline definition instead of reconstructed by whoever reads the table. Here is how the data product behind it is structured, what it exposes to whoever queries it, and where every tile on the dashboard comes from.
Imagine opening your receivables view at eleven in the morning and knowing it is current. Not current as of last night's close. Current as of the invoice that posted four minutes ago and the payment that cleared while you were reading the screen.
Now imagine the thing that makes it correct not being a pipeline somebody tuned, but a block of code you can read: this table's key is these three columns, order its changes by this clock, and when the change type is a delete, delete the row.
That is what an accounts receivable data product on Databricks can be, and it is less about speed than it sounds. What makes it work is that everything underneath is structured so a single row can be corrected in place at the moment the transaction that changes it happens. Get the structure right and freshness is something you schedule. Get it wrong and no amount of streaming saves you, because you will simply be wrong faster.
So this goes from the bottom up, using the Account Receivable Visibility data product as the worked example. Sources, layers, what it publishes, and where every tile on the dashboard comes from.
The whole shape on one page. Everything below is a closer look at one of these four columns.
The shape of the flow: SAP, then Kafka, then Databricks
Three hops, and it is worth stating them plainly before anything else, because everything after this depends on the shape.
SAP to Kafka. This hop starts with a definition, not with code. In the One Connect SAP Data Modeler you declare what a data product is, and the declaration is the deliverable: which SAP tables belong to it, how they join, and what every field is called on the way out.
That definition ships as a pair of spreadsheets, which sounds unglamorous and is the point, because a spreadsheet is a thing a functional consultant can read and argue with. For this product the first one declares the sixteen entities: source table, join fields on both sides, join type, its level in the hierarchy, a business alias, and any filter. VBAK joined to VBAP on VBELN, inner, level two, filtered to order type OR. The second declares 1,203 fields across those entities, and for each one the source table and field, the business name it is published under, whether it is a key, whether it is selectable, and a description. VBRK.VBELN becomes bill_doc, marked as key.
So the join topology and the field semantics are declared once, in a file, before anything streams. What the business events then publish to Kafka is that declaration made live. One topic per source table, created as needed, with schema registry entries and schema evolution handled at that boundary. The precise capture mechanism is a later section; for now what matters is that Kafka is where SAP data lands, not where it is queried.
And you rarely start from an empty modeler. The Onibex Marketplace lists 159 pre-modeled data products across eleven SAP modules, 97 built from tables and 60 from CDS views. The sixteen this product needs come from that catalogue, which is why the interesting work in a receivables project is the last mile rather than the first.
Kafka to Databricks. From those topics the data lands in bronze Delta tables in Unity Catalog, one per SAP source table, and everything above bronze is built by a declarative pipeline that reads the change data feed of the table below it. No orchestrator deciding what changed, no high water mark column somebody has to maintain, no nightly full reload standing in for incremental logic. The table's own commit history is the input.
Databricks to whoever asks. A dashboard, a warehouse, a CRM, a machine learning workload. They query the top layer and never touch SAP. The dashboard that ships with this product is a single Power BI report, and the rest of that list is whatever else in the organization can already read a table.
The reason to separate those three hops rather than build one pipeline from SAP to a report is that the middle hop is where the meaning gets added, and it is the part you want to write once and reuse. Which brings us to how the layers are organized.
The medallion architecture, and what sits in each layer
The middle hop follows the medallion architecture, bronze to silver to gold, and the useful way to read it here is as a count that shrinks: sixteen, then seven, then eight, then two.
Sixteen foundational data products, seven business data products, eight gold entities, five kinds of consumer.
Bronze holds sixteen foundational data products. Raw SAP tables grouped into the smallest units that mean something on their own. VBAK, VBAP and VBFA as the sales order. VBRK and VBRP as the billing document. ACDOCA, BSID, BSAD and BSEG as the journal entries. KNA1, KNVV and KNVP as the customer. PA0000 and PA0001 as personnel. MARA and MARM as the material. TCURR, TCURF and TCURV as currency. Plus eight small ones for organizational structure: sales organization, distribution channel, sales division, sales group, customer group, payment terms, region texts and the Nielsen indicator.
Those sixteen come from four different areas of SAP. Finance contributes three, order to cash four, human resources one, and organizational structure eight. That ratio is the first thing worth noticing: half the sources in a finance data product are not financial. They are the semantics that make a finance number readable by somebody who does not know SAP.
Silver holds seven business data products. One per business concept: journal entries, sales billing, sales order, currency, personnel, material master, customer master. Sixteen becomes seven because the eight organizational structure entities are not concepts in their own right, they are descriptions that get attached later, and the two receivables entities, open items and cleared items, collapse into a single journal entries product.
Gold holds eight entities. The consolidations, the two unit of measure normalizations, the two summaries, and the two that consumers actually query.
Those last two are the product's real surface. Everything else is scaffolding, and the discipline of the medallion pattern is that the scaffolding is visible and inspectable rather than buried inside one query nobody wants to open.
Notice who is on the top row of that diagram. Not only the finance team. A CRM, an operational data store, a generative AI or machine learning workload. The same two contracts serve all of them, which is only possible because they are contracts and not reports.
Two streams, because exposure is two questions
The pipeline runs two streams that never merge. That is deliberate, and it is the structural decision that matters most.
The finance stream answers what has been invoiced and not paid. It is anchored on ACDOCA, the Universal Journal line items, joined to BSEG on document, fiscal year and line item, and to BSID, open customer items, on company code, document, fiscal year, line item and customer.
Then the join that is worth stopping on. BSAD, cleared items, joins on ACDOCA.BELNR equal to BSAD.AUGBL, and on the line item. AUGBL is the clearing document. An invoice does not become paid because a flag flips; it becomes paid because a second document appears and points back at the first. Payment behavior is not a state on the invoice, it is a separate document joined on a field most people never look at.
The sales stream answers what has been committed and not yet invoiced. It is anchored on the sales order: VBAK for the header, VBAP for the items, VBFA for the document flow, VBPA for the partners, VBKD for the business data. Those are the same five tables I wrote about when describing what an SAP data product is, and this product consumes that data product rather than rebuilding it.
The sales stream ends with a definition of an open order written as three exclusions, and on Databricks you can see where the three flags come from rather than only the filter that uses them. Each one is derived from the subsequent document category in VBFA: category M sets the invoice flag, category P sets the deposit flag, category O sets the credit note flag. Everything already billed, prepaid, or credited then comes out. What remains is business the company has committed to and has not yet turned into a receivable.
Two streams, because credit exposure is not the receivables balance. It is what the customer owes plus what they are about to owe, and the second part lives in Sales and Distribution rather than in Finance.
What the product exposes: contract one
The finance stream ends in a table published at one row per journal entry line. The key is client, ledger, document number, line item, fiscal year, and company code.
What a consumer gets on that row, grouped by what it is for:
Identity and posting. Document number, line item, fiscal year, fiscal period, posting date, account, cost center, account type.
The billing link. Billing document, billing date, net value from the billing item, billing type. This is how an accounting line traces back to the invoice a customer actually received.
Receivable state. Due date, payment terms, cleared amount, clearing document and clearing date from the open items side, and the same four again from the cleared items side. Both sides are on the row, which is what makes a single query able to compare outstanding against collected without a second pass.
Measures. Quantity in the material's base unit, and amount in group currency.
Dimensions with their descriptions. Sales organization and its text. Distribution channel and its text. Division and its text. Plant. Customer number, name, city, country, region and the region's text. Customer group and its text. Sales group and its text. Four additional customer group fields. Nielsen indicator and its text.
Ownership. The personnel number of the sales representative.
Every code arrives with its word. That is not decoration; it is the difference between a table a finance analyst can query and a table that needs a lookup sheet.
What the product exposes: contract two
The sales stream ends in a second table, published at one row per open sales order item. The key is the sales order and the item.
The columns are narrower, because an open order has less to say than a posted document: order date, fiscal year and month, order type, sales unit, material, net value, currency, and the three status flags described above.
But the dimensional half is nearly identical to contract one. The same sales organization and text, distribution channel and text, division and text, plant, customer with name, city, country, region and text, the same customer group fields, the same sales representative.
That repetition is the design, not an accident.
Why the dashboard has one filter bar
The dashboard filters on company code, plant, sales organization, sales group, country, and region. Six filters, applied to both halves of the picture at once.
That only works because both contracts carry the same dimensional spine. Filter to a region and the invoiced-and-unpaid number and the committed-but-not-invoiced number both narrow, in the same way, resolving the same code to the same word.
If the two streams had been enriched independently, this is exactly where it would break. One would resolve region from the customer master and the other from the sales document, and a regional exposure figure would stop reconciling with the sum of its parts. The eight organizational structure sources are shared for that reason.
What the dashboard visualizes, and where each tile comes from
The dashboard ships as a single Power BI report, built directly on the two gold entities. It is a starting point, and a starting point only needs to exist once, because the first thing any finance team does with a delivered dashboard is change it.
Six filters across the top, six tiles, three visuals and the ledger. Every element resolves to a column in one of the two gold entities. Figures shown are from a demonstration environment.
What matters more than the tool is that nothing on that page needs a join. The report reads two tables and does arithmetic, which is what a data product is supposed to leave for the consumer.
Total receivables and overdue receivables both resolve to the open item amount. The difference between them is a comparison of the due date against today, which is why the due date and the payment terms travel on the same row as the amount.
Percent overdue is the ratio of those two, and it is the tile most likely to be wrong for structural reasons rather than arithmetic ones. Getting it right depends entirely on which date the product treats as the due date.
Days sales outstanding comes from the two dates the product ships rather than from a column named DSO: the posting date and the clearing date. The product publishes the dates. The measure is defined in the dashboard layer, which is the correct place for it, because two finance teams will define it differently and neither definition belongs baked into a pipeline.
Cash collected reads the cleared side: clearing date and cleared amount, from the BSAD half of the row.
Predicted cash is the one tile that needs both streams. Receivables coming due in the window from the finance stream, plus open orders expected to bill from the sales stream. A tile built on the finance stream alone would forecast only the money already invoiced, which understates the pipeline by exactly the amount of business in flight.
Aging analysis buckets the open items by how far past due they are. Worth being precise about what the product hands over here: it ships the due date and the payment term day count, not the buckets. The bucket boundaries are a business decision and they live in the reporting layer.
Top customers is the customer name and the amount, ranked. The name is on the row because the customer master is one of the sixteen sources.
The invoice ledger is the underlying grain made visible: document number, customer, name, posting date, due date. It is the same rows the tiles aggregate, which is why a number on a tile can be clicked down to the documents behind it.
How the data reaches Databricks
Onibex One Connect captures changes in SAP ECC and SAP S/4HANA through application-layer business events (RAP, BOR, BTE, PPF), pushed outbound over an HTTP RFC destination of Type G to the Smart Gateway, and streamed to Apache Kafka, Confluent Cloud, Databricks, Snowflake and other targets. Nothing is read from the database layer. Nothing is pulled.
That is the first hop, and for receivables the relevant property of it is not raw speed. It is that the update is triggered by the transaction rather than by a schedule, because a schedule does not know when anything happened. It only knows what time it is.
The second hop, Kafka into the medallion layers, is where the more common misreading happens. Bronze, silver and gold are frequently built as scheduled batch jobs, one layer waiting on the one below it, which is how a three-layer architecture turns into a three-window delay. Which brings us to the part of this that is specific to Databricks.
Where the change semantics are actually declared
Every silver and gold table in this pipeline is built the same way, in two moves.
The first move reads the change data feed of the bronze Delta table, not the table itself:
def cdf_reader(table):
return (
spark.readStream.format("delta")
.option("readChangeFeed", "true")
.option("startingVersion", "0")
.table(table)
)
What comes back is not rows, it is changes: each one carrying a _change_type of insert, update or delete, a _commit_version and a _commit_timestamp. The Delta table's own commit history is the change feed. Nothing had to be added to the source to produce it.
The second move turns that stream of changes back into a current-state table, and this is the declaration that matters:
dlt.apply_changes(
target="acdoca_cur",
source="acdoca_cdf",
keys=["belnr", "docln", "gjahr"],
sequence_by="_commit_timestamp",
apply_as_deletes="_change_type = 'delete'",
ignore_null_updates=True,
)
Four statements about the data, in four lines, on the table they apply to.
keys is the declared grain. ACDOCA is unique on document, line and fiscal year. BSID and BSAD need five columns, company code, document, fiscal year, line item and customer. VBAK needs one, VBELN. VBAP needs two. VBFA needs three. That is not configuration, it is the grain of the business object written down where the code that maintains it can be checked against it.
sequence_by is the clock that decides which version of a row wins when two changes arrive for the same key. This is the argument that quietly decides whether your table is right.
apply_as_deletes is the line that makes a delete a delete. Without it, a removed row in SAP leaves a row behind in silver forever, and nothing about that failure is visible, because the evidence of the problem is exactly the record that stopped arriving. I have written about the same failure happening on a different stack for the opposite reason, and the lesson generalizes: a pipeline that cannot express a delete will never tell you it has one.
ignore_null_updates keeps a partial update from blanking fields it did not carry.
The result is what the code calls a Type 1 snapshot: current state, no history. Those _cur tables are what the silver joins read, which means the join is always operating on one committed version of each source row rather than on a stream of changes it would have to reconcile itself.
The argument on clocks, and an honest inconsistency
Two things about that declaration deserve more than a passing mention, and the second one is a defect rather than a design.
The first is what sequence_by actually chooses. Ordering by _commit_timestamp orders changes by when the lakehouse committed them. Ordering by an event timestamp carried from the source orders them by when the business said they happened. Those are the same thing only while delivery is perfectly ordered, and the whole point of putting Kafka in the middle is that delivery is not guaranteed to be. If two updates to the same sales order are produced in one order and committed in the other, commit time confidently applies the wrong one last.
The second is that this pipeline does both. The finance tables sequence by _commit_timestamp. The sales order tables sequence by event_ts, the timestamp the business event carries out of SAP. Two clocks, in one pipeline, on data that ends up on the same dashboard. Each is defensible on its own; having both without a stated reason is not, and it is the kind of thing that only ever surfaces as a number nobody can reconcile. It is on the list to settle, and I am naming it here rather than writing around it.
There is also a naming detail worth knowing before you copy any of this. Databricks has renamed Delta Live Tables to Lakeflow declarative pipelines, and apply_changes is now create_auto_cdc_flow, with the same signature. The dlt module and the old function names still work and the code above still runs, but new pipelines should use from pyspark import pipelines as dp and the current names.
Where the freshness comes from, which is not the table
This is the honest difference from the same product on Snowflake, and it is worth being direct about rather than claiming parity.
On Snowflake, the freshness contract is a property of the table. Every dynamic table declares TARGET_LAG = DOWNSTREAM and defers to the one consuming it, and the final entity declares an actual target, one minute, on the object that answers the question. You can read the intended freshness of the whole pipeline out of one table definition in version control.
On Databricks there is no such line in the table definition. What decides how current the gold tables are is how the pipeline runs: continuous mode, where the pipeline stays up and processes changes as they arrive, or triggered mode, where each run processes what has accumulated and stops. That is a property of the pipeline, not of the table, and it is set where pipelines are configured rather than next to the code a reviewer reads.
The trade is real in both directions. Snowflake makes the freshness target reviewable and puts the incremental mechanics inside the service, where you accept what the planner decides to maintain incrementally. Databricks makes the change mechanics reviewable, which is what the entire section above is about, and leaves freshness to the run configuration.
Neither of those is better. They put the reviewable part in different places, and if you are choosing between them, that is the question to ask rather than which one is faster.
The same contracts on other engines
Worth saying plainly, because it is the reason any of the above is reusable: the two contracts are the product, and Databricks is one way to maintain them.
The same sixteen sources, the same grain and the same column names also ship as Snowflake dynamic tables on incremental refresh, as Flink SQL jobs with an upsert changelog over compacted Kafka topics, and as PostgreSQL layers maintained by stored procedures. What differs across them is where state lives and how the incremental update is expressed. A consumer querying the gold table cannot tell which one is running underneath, and that is the point of publishing a contract instead of a pipeline.
Databricks is the one worth writing up in this much detail because it is the one where the change semantics are written out in full, in code, next to the table they govern. The grain, the ordering clock and the delete rule are four lines somebody can review in a pull request rather than behavior you have to infer from a diagram.
Where the product stops
Four things are deliberately not in it, and a pre-modeled data product that pretends otherwise is worse than one that draws the line.
Thresholds. Nothing in the product decides that a customer at ninety percent of limit gets released and one at ninety-five does not.
Bucket and metric definitions. Aging boundaries and the DSO formula belong to the reporting layer, for the reason given above.
Currency conversion parameters. The gold sales layer converts to USD at exchange rate type M, matched on the document date, with returns negated. The target currency, the rate type and the date the rate is matched on are all choices written into the transformation. All three change the number on the dashboard, and all three need an owner.
History. The current-state tables are Type 1. The change feed that fed them retains only as far back as the source table's history does, so if you need to answer what this row said last quarter, that is a Type 2 design and a different conversation.
What the structure does guarantee is narrower and more useful than a claim about speed. Every tile traces to a column, every column traces to a source, the invoiced half and the committed half of exposure filter the same way, and the rule that decides what happens to a row when SAP changes it is four lines somebody chose rather than behavior nobody wrote down. That is what makes the six tiles worth looking at.
Technical reference: the deployment artifacts and integration steps are documented in oneconnect-docs. The Finance data products this one consumes are listed in the Onibex Marketplace.
Josue Ramirez works on data and AI at Onibex. More at onibex.com.
