Is the serverless budget/usage feature officially broken for certain serverless job "types"?

[excavator-matt]

As we all know, the main method of cost attribution for serverless compute is with budget (sometimes called usage) policies (Public Preview). This is a new paradigm compared with the tagging of both classic compute and serverless SQL warehouses.

The Jobs UI further makes the clarification we should not confuse tags for organising with cost tags. ("Tags are key-value pairs that you can use to organize your jobs.")

However, I have found three things in Databricks that don't make sense based on this.

1. For jobs created with asset bundles and its `tags` parameter, the organisation tags actually appear in the cost break down usage dashboard. Isn't this a bug? How does this interact with potential budget_policy_id tags? When I tried this a couple of months ago, the tags did not appear until I made them a part of a budget policy.

2. It does not appear to be possible in the UI to set a budget policy for certain serverless job types. For most serverless types, the usage policy show up in the settings menu above tags and below Environment/Compute. However it doesn't even show up as blank for serverless jobs created with asset bundle, ClickOps jobs that simply wrap Salesforce managed ingestion or Terraform jobs that wrap declarative pipelines. I am not sure what the rule is here.

3. The tag UI element in the budget policy seems broken as show blank even it the tags are set properly.

Are you scraping budget policy or how should we think about cost tagging serverless processes going forward?

[excavator-matt]

It might be good to clarify what Databricks means by cost tag. This is defined in the provided dashboard Account Usage Dashboard and specifically the chart Usage Analysis: Tag Matching. This chart uses the dashboard dataset usage_analyze_tags. The query is convoluted, but it boils down to custom_tags column in the table system.billing.usage.

[stbjelcevic]

hi @excavator-matt ,

The mental model that budget policies are for cost and tags are for organization is outdated. For serverless workloads, both job-level tags and budget policy tags flow into system.billing.usage.custom_tags, with budget policy tags winning on key conflicts. The Jobs UI text describes intent but understates the billing reality.

1. DAB job tags appearing in custom_tags without a budget policy. By-design, not a bug, and not DAB-specific. Tags set on a serverless job propagate to custom_tags regardless of whether the job is created via UI, API, DAB, or Terraform. The DAB job schema also accepts budget_policy_id at the job level if you want admin-controlled tags layered on top, see the DAB resources reference. One caveat: jobs/submit one-time runs don't propagate tags, only jobs/create and jobs/update. If you saw different behavior earlier this year, propagation has been consistent for some time now.
   
   
2. Budget policy field missing from UI for some serverless job types. Three different reasons depending on the case. 
   
   For DAB-deployed jobs, the UI hides most settings (including budget policy) because DAB jobs are deployed in UI_LOCKED mode by design. See this KB article. Set budget_policy_id in your bundle YAML.
   
   For Terraform-deployed jobs that wrap a Lakeflow Declarative Pipeline, you're hitting a documented limitation on top of the IaC UI lock: per the budget policies docs, "Pipelines triggered by jobs do not inherit the job's serverless usage policy. Users must set the pipeline's policy." Configure the policy on the pipeline directly.
   
   For ClickOps-created managed ingestion jobs (Salesforce, etc.), the budget policy selector has known display gaps in the ingestion wizard context. Workaround: open the underlying ingestion pipeline and set the policy there.
   
   Net rule: when a job's compute is actually a wrapped pipeline (DLT, managed ingestion), the budget policy belongs on the pipeline, not the job. When the job itself is serverless and IaC-managed, set it in YAML/HCL.
   
   
3. Tags element inside Budget Policy showing blank when tags are set. Display bug, billing attribution is not affected. The renderer keys off the job's explicit settings.budget_policy_id field, but certain UI patches can leave that field empty even though effective_budget_policy_id is still set under the hood. Tags continue to land in system.billing.usage.custom_tags correctly. Workarounds: re-select the policy in the dropdown to repopulate the explicit field, and verify directly with:

SELECT custom_tags, usage_metadata
FROM system.billing.usage
WHERE usage_metadata.job_id = '<job_id>'

A related and documented limitation: budget policy tags also don't render on the Jobs list view, only on the Job Details page (same docs page, "Known limitations").

On the broader question, how to think about serverless cost tagging. A "cost tag" in Databricks is anything that lands in system.billing.usage.custom_tags. For serverless that column is the union of budget policy tags (admin-set, win on key conflicts), job-level tags (practitioner-set via UI/API/DAB/Terraform tags field), pipeline tags (set on the DLT or managed ingestion pipeline), and inherited workspace tags. Treat the two main mechanisms as complementary: budget policies for governed admin-controlled attribution that practitioners can't override (cost center, department, environment); job tags for practitioner-managed metadata that also shows up in billing as a bonus (project, team, ticket-id); pipeline-level policies for anything wrapped by a job. The tag attribution docs cover propagation rules end-to-end if you want the full read.

[excavator-matt]

Thank you for your reply! This adds clarity to my confusion regarding serverless tagging. This information would have helped a lot if it was included in the official documentation. You should considering sending the technical documentation team your post. I don't think it was clear that:

1. Job level tag can be used for cost tagging as an alternative to budget policy on serverless, but that it isn't recommended

2. That budget policy takes precedence on key value conflict. (Perhaps I could have experimented my way to this insight, but I rather have Databricks state the behaviour)

I should also add that AWS is moving in the other direction. From cost tags being whatever tag you want, to limiting to some reserved tags. I think this was change for the better as the poor select filter in the usage dashboard now gets overrun in Databricks and you don't risk different version of the same key (env vs Environment).

Regarding the second sub topic of Budget policy field missing from UI. I am personally using Terraform, but I think we should allow our colleagues to try asset bundles and ClickOps ourselves. You say it is more a display gap and limitation, than a bug, but could you ask them to reconsider their design here? It would be nice to quickly check if the job has a budget_policy or not in the UI without CLI or YAML. It somewhat makes sense that you can't modify the asset bundle, but I think it would be better if it stated read only its current value in the same format as the rest of the jobs. Then you wouldn't have to think of all these conditional logic on when a budget is present in the UI or not.

Regarding Tags element inside Budget Policy showing blank, the tags now appear. I don't know if the issue went away or if it only happens on some conditions.