How are you attributing serverless costs back to individual jobs and teams?

[Islam_hoti]

Hi everyone,

We moved most of our workloads to serverless over the last few months and the performance side has been fine. The part we did not plan for is cost attribution.

On classic compute this was easy. One cluster per team, tags on the cluster, and the billing table told us who spent what. With serverless there is no cluster to tag, so the picture is much blurrier. We can see total serverless DBUs going up, but when finance asks which team or which pipeline caused a jump, we do not have a clean answer.

What we have tried so far. We use the system billing tables and group by the job and pipeline identifiers where they exist, but a lot of usage lands in buckets we cannot easily map back to an owner, especially interactive notebook usage and ad hoc SQL. Budget policies help enforce tags going forward, but they did not solve the retroactive picture and they only cover some resource types.

So a few questions for people further along than us.

Are you using budget policies with mandatory tags, and did that actually give you clean attribution, or did you still end up with a large unattributed bucket?

How are you handling interactive notebook usage, where the person running it is not tied to any job? Do you charge that back to individuals, absorb it centrally, or restrict it?

Has anyone built anything on top of the system tables that they would consider genuinely reliable for chargeback, as opposed to directionally useful?

And the honest one: did anyone decide the attribution problem was not worth solving and just accepted serverless as a shared central cost?

Would really appreciate hearing how other teams landed on this, including the approaches that did not work.

Thanks.

[balajij8]

@Islam_hoti 

Databricks serverless compute usage is attributable through the system billing tables and it requires a different approach than legacy cluster based tagging. system.billing.usage table captures serverless usage with usage_metadata that includes job_id, notebook_id, user_name, and workspace_id for most workload types. For jobs and pipelines, you can tag the job or pipeline resource itself (not a cluster), and those tags generally propagate to usage records via the custom_tags column. Budget policies with mandatory tag enforcement is good for new resources going forward - jobs, SQL warehouses and pipelines. Note that they don't retroactively fix untagged historical usage or cover all interactive scenarios. The unattributed bucket typically comes from ad-hoc notebook runs and SQL queries where the resource (job/warehouse) wasn't tagged, but you can still pivot to user-level attribution using identity_metadata.run_as or usage_metadata.user_name to charge individuals or their cost centers.

You can use warehouse-level tagging plus user-identity passthrough - create team-specific or cost-center-specific SQL warehouses with mandatory tags, restrict users to their team's warehouse via entitlements and the billing table will tie usage to both the warehouse tag and the individual user for interactive notebook and SQL warehouse usage.

You can choose to absorb some level of exploratory usage centrally (setting a threshold like <50 DBU/month per user) and chargeback sustained usage. The shared central cost approach is good if serverless represents a small percentage of total spend. You can build a chargeback view on system.billing.usage joined to system.access.audit (for resource ownership) and workspace metadata (for team mappings) as standard practice.

[data_pulse]

@Islam_hoti 

We recently solved this use case to attribute the serverless cost. Our approach was not to rely on a single tagging mechanism, but to build a layered attribution on top of Databricks billing system tables.

The actual DBUs/cost always come from system.billing.usage. We enrich those records with the best available team ownership signal.

	Attribution Signal	Example
1	custom_tags	team=pricing, team=risk..
2	Resource identity	job_id, dlt_pipeline_id, warehouse_id, endpoint_id, app_id
3	Resource → team mapping	Known Job/Pipeline/Warehouse mapped to owning team
4	identity_metadata.run_as	Interactive notebook / Databricks Connect user or SPI
5	Platform/shared	DQ Monitoring, Predictive Optimization, networking, etc.
6	Unattributed	No proper ownership signal

Core billing Columns from the Source

SELECT
  workspace_id,
  usage_date,
  billing_origin_product,
  sku_name,
  usage_metadata.job_id,
  usage_metadata.dlt_pipeline_id,
  usage_metadata.warehouse_id,
  usage_metadata.cluster_id,
  usage_metadata.interactive_source,
  identity_metadata.run_as,
  custom_tags,
  usage_quantity
FROM system.billing.usage;

For team attribution we use something conceptually like:

COALESCE(
  custom_tags['sandbox_teamname'],
  custom_tags['hub_teamname'],
  custom_tags['team'],
  element_at(custom_tags, 'group'),
  resource_team_mapping.mapped_team,
  identity_team_mapping.mapped_team
) AS attributed_team

Serverless jobs/pipelines

For Jobs, pipelines and similar resources, usage_metadata is particularly useful:

job_id → owning team
dlt_pipeline_id → owning team
warehouse_id → owning team
endpoint_id → owning team

We maintain an effective-dated resource_team_mapping for historical cost attribution.

Eg:

resource_type | resource_id | mapped_team | billing_start | billing_end
Job | 123456 | pricing | 2026-05-01 | NULL
DLT Pipeline | abc123 | risk | 2026-04-15 | NULL

That also solves an important historical problem: if a resource was untagged previously but subsequently receives a valid team tag, we can retro-attribute its historical billing records to the same resource/team without modifying system.billing.usage.

Interactive/serverless usage:

For usage without a Job/Pipeline/etc we fall back to: identity_metadata.run_as and also usage_metadata.interactive_source.

For eg:

billing_origin_product = INTERACTIVE
interactive_source = DATABRICKS_CONNECT
run_as = user/service-principal

How we implemented tagging:

• Ownership tags such as team were added directly to workflow definitions in code base, making ownership part of deployment.
• For classic compute, cluster policies, pool tags, and cluster custom tags were used to carry team ownership.
• For serverless Jobs and pipelines, budget policies were deployed through Terraform and referenced through budget_policy_id. Their tags then flow into subsequent billing records under system.billing.usage.custom_tags.
• For SQL warehouse, the ownership tags were provisioned as part of terraform. Can configure manually too, to edit and assign tags.

Budget Policies

Budget/serverless policies help significantly going forward because their tags can propagate into billing usage. But we found they shouldn't be considered the entire attribution solution:

Policy/resource tags
        ↓
system.billing.usage.custom_tags
        ↓
Resource mapping
        ↓
Identity mapping
        ↓
Platform/shared
        ↓
Unattributed

Platform costs

We also deliberately don't force everything onto teams. Things such as:

DATA_QUALITY_MONITORING
PREDICTIVE_OPTIMIZATION
NETWORKING
FINE_GRAINED_ACCESS_CONTROL

could represent Databricks managed/platform activity rather than a team-owned. Those can be kept as Platform/Shared rather than inventing an attribution.

Summary: we didn't accept all serverless spend as a central cost. We attribute what can be reliably attributed using tags → resource IDs → identity, explicitly retain platform/shared costs, and expose whatever remains genuinely unattributed.

I am working on a detailed blog post to cover comprehensively, but the above content covers most parts of it.

[Khasim_1 · Accepted solution]

Hi @Islam_hoti,

This is a classic "Day 2" problem for any platform team shifting to serverless. You’ve hit on the tension between the architectural benefits of serverless (no cluster management) and the organizational requirement for chargeback granularity.

We have found that System Tables are the baseline, but you are right—they are "directionally useful" rather than "finance-perfect" without a strict governance policy.

Here is how we’ve approached the "Attribution Gap":

1. Mandatory Tags & Budget Policies: Budget policies are essential, but they are a "gate," not a "retroactive fixer." We treat them as a non-negotiable architectural standard for all new assets. If an asset isn't tagged, it doesn't move to production. This stops the "unattributed bucket" from growing.
2. The Interactive Notebook Dilemma: We stopped treating ad-hoc usage as a "shared cost." We now restrict interactive serverless usage in PROD environments. For development, we allocate a fixed "Innovation Budget" per team that is absorbed centrally. It’s cleaner to budget a lump sum for R&D than to try and bill an individual for 45 minutes of DBU consumption.
3. The "Reliable Chargeback" Layer: We actually built a custom logic layer on top of System Tables. We map job_id and pipeline_id to a mapping table (which we manage in a separate config file) that links these IDs to team_cost_centers. If a job is orphaned or lacks a tag, it defaults to a "General Platform" bucket, which we review monthly with department leads. It’s not automated "perfection," but it makes the conversation with Finance much easier.

The "Honest" Take: To your last question: Yes, we did decide that at a certain point, the cost of "perfect" attribution exceeds the value of the chargeback. We aim for 90% attribution accuracy. The remaining 10% (the "noisy neighbor" or the "forgotten workspace tag") is treated as a tax for using a shared, high-performance platform.

Has anyone else found a way to automate that last 10% of attribution, or are we all just living with the "Platform Tax"?

[davidwarner344]

We found mandatory tags help a lot for jobs, but interactive usage still needs identity-based attribution. Rather than chasing 100% accuracy, we report team-owned usage plus a small shared/ad-hoc bucket. A simple layer over the system billing tables has been enough for chargeback without trying to reconstruct everything retroactively.

[ivanvyd]

@Islam_hoti the most defensible model is to separate measured attribution from allocation policy instead of forcing every DBU into a team bucket.

1. Exact workload attribution. Use system.billing.usage as the ledger. For serverless jobs, group by workspace_id, usage_metadata.job_id, and usage_metadata.job_run_id, then sum every matching record because one run can emit several billing rows. Join system.billing.list_prices on cloud, SKU, and effective dates. Map the workload to a team with serverless usage policy tags in custom_tags.
2. Interactive notebooks. Use identity_metadata.run_as with usage_metadata.notebook_id and notebook_path, then map that identity to an effective-dated team or cost center. This supports historical allocation, but it is not retroactive tagging: existing assets are not automatically assigned policies, and policy changes affect only new usage.
3. Ad hoc SQL. Billing is exact at warehouse level through usage_metadata.warehouse_id. Use system.query.history fields such as executed_by, query tags, and duration to explain who drove activity. Any per-user split of shared warehouse DBUs is an allocation rule, not a measured per-query bill, so disclose the rule.
4. Shared costs. Keep managed platform activity and records with no defensible owner in explicit Shared and Unattributed buckets. Track those percentages and remediate missing ownership; do not silently spread them across teams.

“For accurate job cost tracking, Databricks recommends running jobs on dedicated job compute or serverless compute” Databricks documentation

For finance, publish separate totals for exact workload, identity-mapped, allocated SQL, shared, and unattributed spend. That gives clean forward attribution without pretending the historical record is more precise than it is. I would centralize only the genuinely shared bucket, not all serverless spend.