Every IT function I've worked with eventually asks the same question, and can never answer it cleanly and clearly: "What did each team actually cost us last month?"
When speaking with the platform team at a large UK bank, the realisation was that the blocker wasn't that the data was missing, or even scattered. It was all sitting right there, in one big pile. Every cost row was captured. The problem was that none of the costs were marked. Nothing on a row told you which team it belonged to — so you could see the total to the penny and still have no idea who to hand it to. A platform function lives or dies on being able to show teams what they cost, and this one was staring at the whole bill unable to split it.
So we fixed it — and the fix wasn't more data, it was a name against every row. Here's the pattern, and it lives entirely in SQL over governed tables.
The principle: a cost you can't attribute is a cost nobody owns
Here is the step-by-step of how we did it:
- The work splits into two stages: first allocate the whole bill across every cost type, then go inside the Databricks spend and attribute it to the pipeline, dashboard or team that caused it. A note upfront: nothing here is cloud-specific — the same pattern works whether your bill comes from AWS (CUR), Azure (Cost Management) or GCP (billing export). Normalise them all to a common shape (FOCUS is the emerging standard) and the rest is identical.
- Step 1: Land all cost sources in one governed table. Pull your cloud provider's billing export and the Databricks billing/usage data into one schema, one grain (one row per resource per day). Keep a cost_type column so you can always see the split: Storage · Compute (VMs) · Databricks (DBUs) · Application / platform services.
- Step 2: Agree one tagging taxonomy across all of it. Decide the allocation key (team / cost centre) once and apply it to every cost type — cloud resources and Databricks workloads. Enforce it going forward with cloud tag policies and Databricks compute/budget policies so the tag lands in billing automatically.
- Step 3: Attribute every row to a team. Prefer the tag; fall back to a resource→owner mapping; label the rest honestly.
- SELECT cost_type,
coalesce(tags.team, owner_map.team, 'UNALLOCATED') AS team,
sum(cost) AS cost
FROM finops.all_cloud_costs c
LEFT JOIN finops.owner_map ON c.resource_id = owner_map.resource_id
GROUP BY 1, 2;
- Step 4: Roll up cost-type × team. Now you can tell any team its full footprint — "here's your Storage, your VMs, your Databricks, your app services" — not just one slice.
Stage 2 — go inside the DBU spend
"Databricks: £X" isn't actionable. Within it, cost is driven by three different things, and each needs its own tag.
5. Step 5: Tag every unit of Databricks work. Put a team/owner tag on each pipeline/job, each SQL warehouse, and each dashboard/app. Job and warehouse tags flow into the usage records as custom tags — that's your primary key for pipeline and job DBUs.
6. Step 6: Attribute shared-warehouse DBUs to the actual consumer. Dashboards and ad-hoc/department queries usually share a warehouse, so tags alone won't split them. Join the warehouse's usage to query history and apportion its DBUs by who actually ran what.
-- split a shared warehouse's DBUs across consumers by compute used
SELECT q.executed_by, q.query_source, -- dashboard / user / dept
sum(q.execution_secs) / sum(sum(q.execution_secs)) OVER () AS share
FROM system.query.history q
WHERE q.warehouse_id = :wh
GROUP BY 1, 2;
Multiply each consumer's share by the warehouse's DBU cost for the period.
7. Step 7. Produce the two-level view. Team → (Storage / VM / DBU / App), and within DBU → (which pipelines, which dashboards, which departments' queries). That's the level at which a team can actually change its bill.
The hard part — shared pipelines and data used by many teams
The moment one pipeline ingests data that several teams consume, "who pays?" has no obvious answer. Don't guess — pick a model and state it.
8. Step 8. Find the shared assets. Use lineage to spot any pipeline/table with more than one downstream consumer. Those are the rows you can't simply tag to one team.
9. Step 9. Choose an allocation basis and make it explicit. Common options, roughly fairest first:
- Usage-driven cross-charge — split the pipeline's cost across consuming teams by how much they actually use the output (bytes/rows read, or query compute against the downstream tables, from lineage + query history). Fairest, and it rewards teams that consume less.
- Equal split — divide across named consumers. Simple, defensible when usage is hard to measure.
- Platform-funded shared service — the platform/EDP team owns it centrally as common infrastructure, funded by leadership, not cross-charged. Right for truly foundational data everyone depends on.
10. Step 10: Cross-charge on that basis. For usage-driven, weight the shared pipeline's cost by each consumer's downstream read/compute share:
-- consuming team's share of a shared pipeline, by downstream compute
SELECT downstream_team,
sum(compute) / sum(sum(compute)) OVER () AS charge_share
FROM finops.lineage_consumption -- shared table -> consuming team -> compute
WHERE shared_pipeline_id = :pipe
GROUP BY 1;
11. Step 11: Leave genuinely common ingestion in a named shared-services cost centre that leadership owns — and show it as its own line. Never dissolve it silently into other teams; that's how the total stops reconciling and trust goes.
12. Step 12: Publish the cross-charge with the rule attached. Each consuming team sees not just what it was charged for shared data, but why and on what basis. The rule being visible is what makes people accept it.
It's not easy, it takes time. But doing it fully and properly makes IT department properly in charge and in control
