How to calculate accurate usage cost for a longer contractual period?

gyorgyjelinek — Wed, 20 Nov 2024 10:29:13 GMT

Hi Experts!

I work on providing and accurate total cost (in DBU and USD as well) calculation for my team for the whole ongoing contractual period. I'v checked the following four options:

Account console: Manage account - Usage - Consumption (Legacy): this dashboard covers only the latest 1 year.
Account console: Manage account - Usage - Consumption: this dashboard covers a longer period, but the first 2 months are still missing.
Checking table system.biling.usage: this one is a main datasource for the previous consumption dashboard, but I tried to query it without joining anything for finding the missing 2 months usage - but no success.
Last option I am going to check is the REST API for billable usage logs. https://docs.databricks.com/api/account/billableusage/download But this option seems to be a quite large task based on this documentation: https://docs.databricks.com/en/admin/account-settings/billable-usage-delivery.html#high-level-flow

Is it worth to spend a couple of more hours with option 4. and build an application for retrieving and processing the logs - is there a good chance to find the older usage logs here?

Is there any other method worth a try?

Re: How to calculate accurate usage cost for a longer contractual period?

mark_ott — Tue, 04 Nov 2025 12:45:48 GMT

Based on your description, the REST API for billable usage logs (Option 4) is likely the most comprehensive and reliable method for retrieving usage and cost data for the full contractual period, including potentially the missing first two months. The API is designed to provide detailed usage data (DBU and cost) across the whole account, and unlike the console dashboards, it is not limited by front-end pagination or time window restrictions. However, it does involve more upfront work for coding, authentication, and parsing log files, as you noted.

REST API Coverage for Historical Data

The REST API for billable usage is the recommended solution for full historical coverage, provided that Databricks has retained logs for your contractual period.
If your workspace or account was configured to deliver billable usage logs from contract start, you will likely be able to retrieve those historical records.
If you see gaps, check with Databricks support or your account manager regarding log retention policies and whether older logs are accessible.

Feasibility of Building a Retrieval Application

Building a retrieval and processing application will take several hours (potentially longer if you have complex requirements or need aggregated breakdowns).
If precise, auditable historical cost calculation is required for reporting, reconciliation, or compliance, this is the best investment, as you can automate future cost audits as well.
The API delivers granular CSV records, which can be filtered and processed to calculate DBU and USD totals per contractual period.

Other Methods to Consider

Contact Databricks support: Sometimes support can manually retrieve historical billing/export missing usage records for specific gaps.
Enterprise agreements: If you have an enterprise-level contract, your account team may provide consolidated spend reports that are not available on standard dashboards.
Programmatic access via SCIM or admin APIs: These are less relevant for usage/cost but worth checking depending on your setup.
Audit logs: Rarely useful for DBU/cost, but may offer clues to usage events if all else fails.

Recommendation

If the missing months are critical, and you require complete coverage, pursuing Option 4 (REST API for Billable Usage Logs) is justified, especially if other sources (console, system tables) have documented gaps.
The time spent building the application will pay off with more accurate, flexible, and repeatable cost tracking.
As a backup, reach out to Databricks support simultaneously to see if they can provide access or missing records in bulk—this could save you effort if available.