cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Serverless Compute Cost Monitoring (System Tables)

Hertz
New Contributor II

Hello,

I have developed a dashboard for monitoring compute costs using system tables, allowing tracking of expenses by Cluster Name (user created name), Job Name, or Warehouse Name. However, with the introduction of the new shared serverless compute, particularly with the SKU name "ENTERPRISE_JOBS_SERVERLESS_COMPUTE_US_EAST_N_VIRGINIA," four billing origin products are being generated.

The first product is 'SQL,' which seems to pertain to Delta Live Tables (DLT), as the only data associated with it is a deltapipeline_id. It is unclear why this is categorized under 'SQL.' The second product is 'JOBS,' which includes a job_id. However, there is no job corresponding to this job_id in the data, nor is there a DLT pipeline associated with the pipeline_id.

Could someone provide guidance on how to accurately track and categorize these costs?

Thank you.

3 REPLIES 3

kaiz
New Contributor II

Hi @Hertz , thanks for your question.

For the first case (billing_origin_product = "SQL"), that represents usage of materialized views or streaming tables on serverless DBSQL. Databricks bills for such usage using the serverless SKU.

For the second case, can you please clarify whether these are records where SKU name is "ENTERPRISE_JOBS_SERVERLESS_COMPUTE_US_EAST_N_VIRGINIA", billing_origin_product is "JOBS", and job_id is null? Or is job_id non-null in the system table but you can't find the job?

Please feel free to DM me as well with your workspace ID and the usage dates you're looking at.

lucianamanzonel
New Contributor II

Hi! I´m facing the same problem here. I´m not being able to check where this cost from ENTERPRISE_JOBS_SERVERLESS_COMPUTE_US_EAST_N_VIRGINIA was generated. They have the 

billing_origin_product = 
SHARED_SERVERLESS_COMPUTE but no more information.
Were you able to discover @Hertz ?

augustsc
New Contributor II

Hi! We're also facing the same problem. We don't have any materialized views or streaming tables, but we are still seeing PREMIUM_JOBS_SERVERLESS_COMPUTE_EU_NORTH with billing_origin_product = 

SHARED_SERVERLESS_COMPUTE generated each day at a time where neither users nor workflows are active. We do also need to track this cost.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group