topic Re: Serverless Compute Cost Monitoring (System Tables) in Data Engineering

Serverless Compute Cost Monitoring (System Tables)

Hertz — Wed, 26 Jun 2024 12:53:48 GMT

Hello,

I have developed a dashboard for monitoring compute costs using system tables, allowing tracking of expenses by Cluster Name (user created name), Job Name, or Warehouse Name. However, with the introduction of the new shared serverless compute, particularly with the SKU name "ENTERPRISE_JOBS_SERVERLESS_COMPUTE_US_EAST_N_VIRGINIA," four billing origin products are being generated.

The first product is 'SQL,' which seems to pertain to Delta Live Tables (DLT), as the only data associated with it is a deltapipeline_id. It is unclear why this is categorized under 'SQL.' The second product is 'JOBS,' which includes a job_id. However, there is no job corresponding to this job_id in the data, nor is there a DLT pipeline associated with the pipeline_id.

Could someone provide guidance on how to accurately track and categorize these costs?

Thank you.

Re: Serverless Compute Cost Monitoring (System Tables)

kaiz — Wed, 26 Jun 2024 22:47:25 GMT

Hi @Hertz , thanks for your question.

For the first case (billing_origin_product = "SQL"), that represents usage of materialized views or streaming tables on serverless DBSQL. Databricks bills for such usage using the serverless SKU.

For the second case, can you please clarify whether these are records where SKU name is "ENTERPRISE_JOBS_SERVERLESS_COMPUTE_US_EAST_N_VIRGINIA", billing_origin_product is "JOBS", and job_id is null? Or is job_id non-null in the system table but you can't find the job?

Please feel free to DM me as well with your workspace ID and the usage dates you're looking at.

Re: Serverless Compute Cost Monitoring (System Tables)

lucianamanzonel — Wed, 14 Aug 2024 15:02:06 GMT

Hi! I´m facing the same problem here. I´m not being able to check where this cost from ENTERPRISE_JOBS_SERVERLESS_COMPUTE_US_EAST_N_VIRGINIA was generated. They have the

billing_origin_product =

SHARED_SERVERLESS_COMPUTE but no more information.
Were you able to discover @Hertz ?

Re: Serverless Compute Cost Monitoring (System Tables)

augustsc — Wed, 21 Aug 2024 06:21:22 GMT

Hi! We're also facing the same problem. We don't have any materialized views or streaming tables, but we are still seeing PREMIUM_JOBS_SERVERLESS_COMPUTE_EU_NORTH with billing_origin_product =

SHARED_SERVERLESS_COMPUTE generated each day at a time where neither users nor workflows are active. We do also need to track this cost.