cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Discrepancy between Azure Billing and Databricks System Tables

Danish11052000
Contributor

11 hours ago

Hi everyone,

I am currently building a cost-tracking dashboard, but I’ve run into a major reconciliation issue between the Azure Portal and Databricks.

The Numbers (Current Month):

  • Azure Cost Management Portal: $2,041.79 (Total cost as per specific adb component).

  • Databricks System Tables (Query): $3,365.959 (Calculated using system.billing.usage joined with system.billing.list_prices).

The Methodology: I am joining system.billing.usage with system.billing.list_prices on sku_name and matching the usage_start_time within the price_start_time and price_end_time windows. I am using the pricing.effective_list.default value for the calculation.

The Problem: My Databricks system table calculation cost is HIGHER than what Azure is actually billing us.

  1. Negotiated Discounts: Does system.billing.list_prices reflect Enterprise Agreement (EA) or MACC discounts? Or is it strictly the public MSRP?

  2. Infrastructure Costs: Typically, Azure should be higher because it includes VMs/Storage. Since my Databricks calculation is higher, does this confirm that the DBU "List Price" in the system table is ignoring our contract's negotiated unit rate?

  3. Reconciliation: How are you guys handling this? Are you applying a manual "Discount Multiplier" in SQL, or is there a way to pull the "Actual Billed Price" directly into Unity Catalog?

Any insights from folks who have reconciled these two sources would be greatly appreciated!

0 Kudos
1 REPLY 1

emma_s
Databricks Employee
Databricks Employee

7 hours ago

Hi,

Yes you're correct in your conclusion the Databricks tables just use list price and therefore don't apply any negotiated discounts. They also won't include the underying VM cost when using classic compute. Most of our customers just handle the discount by applying a fixed percentage discount in the SQL.  If you haven't tried it, it's pretty easy to use Genie Code to build a version of the dashboard for you with the discount applied. Some people do engineer pipelines that brings in their cloud billing data into databricks, but in my opinion for most cases this is overkill.

 

Thanks,

Emma

0 Kudos
Announcements