Weโve observed that the system.billing.usage table in Databricks is not capturing job usage data in real-time. There appears to be a noticeable delay between when jobs are executed and when their corresponding usage records appear in the system table.
This is impacting our ability to monitor DBU consumption and perform timely cost tracking and alerting.
Has anyone else encountered this issue recently? If so:
Is there an expected lag for data to be populated in system.billing.usage?
Are there any known limitations or configuration settings that could affect the real-time ingestion of billing data?
Any recommended workaround for real-time usage tracking?
Thanks
Vamsi