topic Re: Databricks Business dashboards - Interactive cluster Total dollar spent in Data Engineering

Databricks Business dashboards - Interactive cluster Total dollar spent

Data_NXT — Wed, 24 Sep 2025 17:22:07 GMT

I'm working on Databricks Business Dashboards and trying to calculate interactive cluster compute time and total dollar spend per workspace.

As per standard understanding, the total dollar spent = Interactive Clusters + Job Clusters + SQL Warehouses.

I'm able to fetch total spend using system, billing, usage and price references from system ,billing ,list prices. However, to isolate Interactive Cluster usage only, is there any Databricks-recommended filter (e.g., cluster type, tags, or system fields) that cleanly segregates Interactive Clusters from Job or SQL clusters?

Appreciate any documented best practices or SQL-level filtering logic you've used successfully.

Re: Databricks Business dashboards - Interactive cluster Total dollar spent

nayan_wylde — Wed, 24 Sep 2025 18:13:55 GMT

@Data_NXT You can look for the following filters.

1. sku_name: STANDARD_ALL_PURPOSE_COMPUTE OR PREMIUM_ALL_PURPOSE_COMPUTE

2. usage_metadata.cluster_id:

Present for non-serverless compute, including interactive clusters.
Absence of job_id or warehouse_id further confirms it's not a job or SQL warehouse.

%sql SELECT workspace_id, usage_date, usage_quantity, sku_name, usage_metadata.cluster_id, identity_metadata.run_as FROM system.billing.usage WHERE sku_name IN ( 'STANDARD_ALL_PURPOSE_COMPUTE','PREMIUM_ALL_PURPOSE_COMPUTE') AND usage_metadata.cluster_id IS NOT NULL AND usage_metadata.job_id IS NULL AND usage_metadata.warehouse_id IS NULL

Re: Databricks Business dashboards - Interactive cluster Total dollar spent

nayan_wylde — Wed, 24 Sep 2025 18:21:20 GMT

Also the system table will not provide you the exact dollar amount that you spend in an interactive compute.

Here is the cost breakdown for running interactive compute:

Component	Description	Cost Source
DBU Cost	Based on workload type and tier	Databricks
VM Cost	Based on instance type and region	Azure/AWS
Storage	Blob + managed disk	Azure/AWS
Networking	NAT gateway, private endpoints	Azure/AWS
Discounts	DBCU, Reserved VM, Spot	Databricks + Azure/AWS

Best Practices for Accurate Cost Attribution
1. Use system.billing.usage table to track DBU consumption.
2. Join with Azure Cost Management data to get VM and infra costs.
3. Apply filters for workload type (sku_name, cluster_id, etc.) to isolate cluster types.
4. Monitor discount application via Azure Reservations portal.

Re: Databricks Business dashboards - Interactive cluster Total dollar spent

Data_NXT — Wed, 24 Sep 2025 19:24:01 GMT

That was really very helpful. Thanks again.