- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
4 weeks ago
Thanks, that was helpful and gave another way to solve this use case.
Instead of maintaining a separate table for object to team mapping table, can now use UC tags as the ownership source with precedence:
table tag -> schema tag -> catalog tag
and join that to system.storage.predictive_optimization_operations_history.
Query shape now looks like:
SELECT
DATE(po.start_time) AS usage_date,
po.catalog_name,
po.schema_name,
po.table_name,
st.tag_value AS team,
po.operation_type,
SUM(po.usage_quantity) AS estimated_dbus
FROM system.storage.predictive_optimization_operations_history po
LEFT JOIN system.information_schema.schema_tags st
ON po.catalog_name = st.catalog_name
AND po.schema_name = st.schema_name
AND LOWER(st.tag_name) IN ('team', 'team_name')
WHERE po.usage_unit = 'ESTIMATED_DBU'
GROUP BY ALL;This gives the PO estimated DBUs by team directly from UC ownership tags. We can extend the same pattern with table and catalog tags as fallbacks.
The only caveat is that these are estimated DBUs, so would still need to reconcile them with system.billing.usage for actual billing/cost reporting.
For DQ Monitoring, found a similar potential path. system.billing.usage exposes schema_id / table_id and system.data_quality_monitoring.table_results exposes those IDs together with the catalog/schema/table names. This should resolve the billed object back to its UC ownership tags without maintaining mapping table.
I haven’t validated it yet as currently don't have access to the DQ monitoring system table, but we’ll test it once access is enabled.
So the attribution path now looks like:
PO: PO operations history → UC object → UC ownership tag → team → reconcile with billing.
DQ: Billing schema_id/table_id → DQ table_results → UC object → UC ownership tag → team