topic Re: Building a Custom Usage Dashboard using APIs for Job-Level Cost Insights in Get Started Discussions

Building a Custom Usage Dashboard using APIs for Job-Level Cost Insights

Sase — Thu, 23 Jan 2025 12:36:20 GMT

Since Databricks does not provide individual cost breakdowns for components like Jobs or Compute, we aim to create a custom usage dashboard leveraging APIs to display the cost of each job run across Databricks, Azure Data Factory (ADF), or serverless environments. If anyone has experience with a similar use case or has implemented such a solution, your insights would be greatly appreciated!

Re: Building a Custom Usage Dashboard using APIs for Job-Level Cost Insights

Isi — Thu, 23 Jan 2025 23:01:16 GMT

Hey!

Databricks recently introduced system tables that provide job cost analysis, which might help achieve your goal without building a custom solution from scratch.

These tables can offer insights into job run costs across Databricks, and you might be able to correlate them with external tools like Azure Data Factory or serverless environments by combining metadata and billing data.

You can check out the official documentation here: Databricks System Tables - Jobs Cost

I hope you find this helpful. 🙂

Re: Building a Custom Usage Dashboard using APIs for Job-Level Cost Insights

Sase — Sat, 25 Jan 2025 13:14:50 GMT

Hey, thanks for sharing the link! I did check out the system tables in the official documentation, but they don’t seem to include the cost of jobs running on SQL warehouses and all-purpose compute. I’m working on a dashboard to get a complete view of all costs, including those.

Re: Building a Custom Usage Dashboard using APIs for Job-Level Cost Insights

Isi — Sat, 25 Jan 2025 14:04:14 GMT

Hey!

You’re right; the system tables might not provide a full breakdown of costs. The approach you take to track costs also depends on the type of cluster you’re using.

If you’re using a Classic or Pro cluster in SQL Warehouse or all-purpose, you’ll likely need to create a custom dashboard— for example, using Grafana. This can pull cost data directly from your cloud provider (AWS, Azure, GCP) to track the compute machine costs and combine it with Databricks-specific costs such as DBU usage and storage.

Hope that helps! 🙂

Re: Building a Custom Usage Dashboard using APIs for Job-Level Cost Insights

Sase — Mon, 27 Jan 2025 12:36:00 GMT

Thanks. Any recommendations on how Grafana can retrieve job, serverless, and SQL warehouse cost data from Azure? Are there specific APIs or other mechanisms available for extracting cost information?

Re: Building a Custom Usage Dashboard using APIs for Job-Level Cost Insights

Isi — Tue, 28 Jan 2025 19:43:11 GMT

Hey,

Yes, I am not Azure expert but, Databricks REST API can help you extract usage data for serverless resources, allowing you to integrate this information into custom dashboards or external tools like Grafana.

On the Azure side, costs related to will appear under the Databricks resource category in Azure Cost Management + Billing. However, Azure does not break down Databricks-managed workloads (like serverless SQL warehouses) into specific line items, as Databricks abstracts the underlying infrastructure. You’ll see an aggregate charge for Databricks usage.

You can configure Azure Cost Management to export aggregated Databricks costs to a storage account or a Log Analytics Workspace. This exported data can then be combined with usage data from the Databricks API to build a more detailed view of costs.

🙂