Based on your description, the REST API for billable usage logs (Option 4) is likely the most comprehensive and reliable method for retrieving usage and cost data for the full contractual period, including potentially the missing first two months. The API is designed to provide detailed usage data (DBU and cost) across the whole account, and unlike the console dashboards, it is not limited by front-end pagination or time window restrictions. However, it does involve more upfront work for coding, authentication, and parsing log files, as you noted.
REST API Coverage for Historical Data
-
The REST API for billable usage is the recommended solution for full historical coverage, provided that Databricks has retained logs for your contractual period.
-
If your workspace or account was configured to deliver billable usage logs from contract start, you will likely be able to retrieve those historical records.
-
If you see gaps, check with Databricks support or your account manager regarding log retention policies and whether older logs are accessible.
Feasibility of Building a Retrieval Application
-
Building a retrieval and processing application will take several hours (potentially longer if you have complex requirements or need aggregated breakdowns).
-
If precise, auditable historical cost calculation is required for reporting, reconciliation, or compliance, this is the best investment, as you can automate future cost audits as well.
-
The API delivers granular CSV records, which can be filtered and processed to calculate DBU and USD totals per contractual period.
Other Methods to Consider
-
Contact Databricks support: Sometimes support can manually retrieve historical billing/export missing usage records for specific gaps.
-
Enterprise agreements: If you have an enterprise-level contract, your account team may provide consolidated spend reports that are not available on standard dashboards.
-
Programmatic access via SCIM or admin APIs: These are less relevant for usage/cost but worth checking depending on your setup.
-
Audit logs: Rarely useful for DBU/cost, but may offer clues to usage events if all else fails.
Recommendation
-
If the missing months are critical, and you require complete coverage, pursuing Option 4 (REST API for Billable Usage Logs) is justified, especially if other sources (console, system tables) have documented gaps.
-
The time spent building the application will pay off with more accurate, flexible, and repeatable cost tracking.
-
As a backup, reach out to Databricks support simultaneously to see if they can provide access or missing records in bulk—this could save you effort if available.
For accurate, contract-period total cost in DBU and USD, the REST API for billable usage logs is your best option for full history, and investing in an application is justified. Also, try Databricks support for potential bulk retrieval or log retention checks.
