topic Re: Do UC table tags propagate to billing for Predictive Optimization and Data Quality Monitoring? in Data Engineering

Do UC table tags propagate to billing for Predictive Optimization and Data Quality Monitoring?

data_pulse — Wed, 09 Sep 2026 15:46:07 GMT

I am looking at a scenario for team-level cost attribution using system.billing.usage and have a tagging gap for Databricks-managed services such as Predictive Optimization and Data Quality Monitoring.

Tried adding a team/ownership tag directly to the underlying Unity Catalog table, expecting that when Predictive Optimization or Data Quality Monitoring operates on that table, the tag would propagate to the corresponding billing usage.

However, it's not the reflected in : system.billing.usage.custom_tags

Usage can be identified with:

SELECT usage_date, billing_origin_product, sku_name, usage_metadata.schema_id, custom_tags, usage_quantity FROM system.billing.usage WHERE billing_origin_product IN ( 'PREDICTIVE_OPTIMIZATION', 'DATA_QUALITY_MONITORING', 'LAKEHOUSE_MONITORING' ) ORDER BY usage_date DESC;

Trying to determine whether the following attribution flow is supported:

UC Table team = "team-a" ↓ Predictive Optimization / DQ Monitoring ↓ Databricks-managed compute ↓ system.billing.usage.custom_tags team = "team-a"

Is propagation of catalog/schema/table tags into billing usage supported for these services?

If not, what could be the recommended way to attribute Predictive Optimization and Data Quality Monitoring costs back to the team owning the underlying table/schema?

We are currently considering a separate object-to-team mapping for cost attribution (Kind of mapping table), but wanted to check whether there is a native/supported mechanism before doing so.

Re: Do UC table tags propagate to billing for Predictive Optimization and Data Quality Monitoring?

stbjelcevic — Wed, 09 Sep 2026 18:38:25 GMT

Hi @data_pulse ,

Short answer: no, that propagation isn't supported. Unity Catalog catalog/schema/table tags do not flow into system.billing.usage.custom_tags for Predictive Optimization, Data Quality Monitoring, or Lakehouse Monitoring. This is by design, not a gap in how you've set things up.

The reason is what custom_tags is scoped to. It only reflects tags applied to compute resources: workspaces, pools, clusters, SQL warehouses, database instances, and serverless usage policies (billing tag docs). Predictive Optimization and Data Quality Monitoring run on Databricks-managed background compute that you don't own or configure, so there's no taggable compute resource for a UC object tag to attach to. Databricks documents the same behavior for materialized views and streaming tables: object-level tags are "not automatically appended to billing records" and have to be joined in manually (materialized view monitoring docs).

What does work today is usage_metadata. For these services it carries the object identifier, schema_id for anomaly detection and table_id for data profiling (billing system table reference). That gives you two ways to attribute the cost back to the owning object.

1. Governed UC tags plus a manual join. If you already apply a team governed tag to your objects, you can join usage to the tag system tables (the same table_tags join the materialized view docs point to) instead of maintaining a separate list. Lower maintenance, since it reuses tags you're already applying.

2. Object-to-team mapping table. Your instinct here is the right fallback. Key it on the schema_id/table_id that appear in usage_metadata so you never have to resolve names:

sql
-- mapping you maintain
CREATE TABLE my_catalog.billing.team_ownership_map (
  object_id   STRING,   -- schema_id or table_id from usage_metadata
  object_type STRING,   -- 'schema' or 'table'
  team        STRING,
  cost_center STRING
);

-- attribution query
SELECT
  u.billing_origin_product,
  m.team,
  m.cost_center,
  SUM(u.usage_quantity) AS total_dbus
FROM system.billing.usage u
LEFT JOIN my_catalog.billing.team_ownership_map m
  ON m.object_id = COALESCE(u.usage_metadata.table_id, u.usage_metadata.schema_id)
WHERE u.billing_origin_product IN (
  'PREDICTIVE_OPTIMIZATION',
  'DATA_QUALITY_MONITORING',
  'LAKEHOUSE_MONITORING'
)
GROUP BY ALL

Both patterns are what teams use in practice for background-compute attribution. There's no native "tag the table, see it in billing" path for these services today, so one of the two joins above is the way to go

Re: Do UC table tags propagate to billing for Predictive Optimization and Data Quality Monitoring?

data_pulse — Wed, 09 Sep 2026 19:39:00 GMT

@stbjelcevic

Thanks for the response and clarification. We are currently relying on usage_metadata IDs as work around mentioned but this adds bit of maintenance overhead in federated team model with many team owned data products.

With materialized views/streaming tables, there is at least an associated compute context that can be used for attribution. PO/DQ is slightly different because the processing runs on Databricks managed serverless where we don't have an underlying compute resource to tag.

Is there anything in the roadmap to support UC object table/schema tags as native attribution mechanism for PO/DQ?

Re: Do UC table tags propagate to billing for Predictive Optimization and Data Quality Monitoring?

Gokul_Pillai1 — Wed, 09 Sep 2026 21:22:12 GMT

Thanks for the follow-up @data_pulse . There is no native propagation of UC catalog, schema, or table tags into system.billing.usage.custom_tags for Predictive Optimisation or Data Quality Monitoring today, and I have not found a public roadmap commitment for that capability.

There are, however, supported attribution surfaces available now:

Predictive Optimization: system.storage.predictive_optimization_operations_history provides the table_id, catalog/schema/table names, and estimated DBUs.
Data Quality Monitoring: system.billing.usage provides usage_metadata.schema_id for anomaly detection and usage_metadata.table_id for data profiling.

For a federated model, the practical approach is a centrally managed ownership dimension keyed by the stable schema/table IDs. That dimension can be populated or validated from your UC ownership tags, but the final attribution remains a post-processing join rather than native billing-tag propagation.

References: Predictive Optimization system table and billable usage system table.

Re: Do UC table tags propagate to billing for Predictive Optimization and Data Quality Monitoring?

stbjelcevic — Wed, 09 Sep 2026 23:53:31 GMT

Fair point, the ID joins do get heavy in a federated model. Two things that help:

For Predictive Optimization specifically, you can skip the billing join entirely. system.storage.predictive_optimization_operations_history already reports catalog, schema, table, and table_id per operation alongside estimated DBUs, so PO cost attributes to a table directly from that table.

To avoid maintaining a mapping table, join to the UC tag system tables (system.information_schema.table_tags, with schema_tags/catalog_tags for rollup) instead of a hand-kept lookup. Each team's own tags in UC become the source of truth, so there's nothing to babysit. The same idea works for billing-only usage via the stable schema_id/table_id in usage_metadata.

On native UC object tag propagation into custom_tags: not supported today, and I can't speak to roadmap here.

Re: Do UC table tags propagate to billing for Predictive Optimization and Data Quality Monitoring?

data_pulse — Thu, 10 Sep 2026 09:46:03 GMT

@stbjelcevic

Thanks, that was helpful and gave another way to solve this use case.

Instead of maintaining a separate table for object to team mapping table, can now use UC tags as the ownership source with precedence:

table tag -> schema tag -> catalog tag

and join that to system.storage.predictive_optimization_operations_history.

Query shape now looks like:

SELECT DATE(po.start_time) AS usage_date, po.catalog_name, po.schema_name, po.table_name, st.tag_value AS team, po.operation_type, SUM(po.usage_quantity) AS estimated_dbus FROM system.storage.predictive_optimization_operations_history po LEFT JOIN system.information_schema.schema_tags st ON po.catalog_name = st.catalog_name AND po.schema_name = st.schema_name AND LOWER(st.tag_name) IN ('team', 'team_name') WHERE po.usage_unit = 'ESTIMATED_DBU' GROUP BY ALL;

This gives the PO estimated DBUs by team directly from UC ownership tags. We can extend the same pattern with table and catalog tags as fallbacks.

The only caveat is that these are estimated DBUs, so would still need to reconcile them with system.billing.usage for actual billing/cost reporting.

For DQ Monitoring, found a similar potential path. system.billing.usage exposes schema_id / table_id and system.data_quality_monitoring.table_results exposes those IDs together with the catalog/schema/table names. This should resolve the billed object back to its UC ownership tags without maintaining mapping table.
I haven’t validated it yet as currently don't have access to the DQ monitoring system table, but we’ll test it once access is enabled.

So the attribution path now looks like:

PO: PO operations history → UC object → UC ownership tag → team → reconcile with billing.
DQ: Billing schema_id/table_id → DQ table_results → UC object → UC ownership tag → team