topic Re: Unexpected SKU Names in Usage Table for Job Cost Calculation in Data Engineering

Unexpected SKU Names in Usage Table for Job Cost Calculation

vziog — Tue, 22 Apr 2025 14:51:33 GMT

I'm trying to calculate the cost of a job using the usage and list_prices system tables, but I'm encountering some unexpected behavior that I can't explain.

When I run a job using a shared cluster, the sku_name in the usage table is PREMIUM_JOBS_SERVERLESS_COMPUTE_EU_WEST for the specific job_id and job_run_id found in the usage_metadata column. I was expecting to see PREMIUM_ALL_PURPOSE_COMPUTE, or possibly NULL, since both usage_metadata.job_id and usage_metadata.job_run_id are NULL.

Additionally, when I run the job on a Job Compute cluster, I see both the expected PREMIUM_JOBS_COMPUTE SKU and the PREMIUM_JOBS_SERVERLESS_COMPUTE_EU_WEST SKU in the usage records.

Could you please help me understand why the PREMIUM_JOBS_SERVERLESS_COMPUTE_EU_WEST SKU appears in each of these scenarios?

Re: Unexpected SKU Names in Usage Table for Job Cost Calculation

Walter_C — Tue, 22 Apr 2025 15:38:39 GMT

Just for confirmation there is no task on this job that has been selected to run with Serverless right?

Re: Unexpected SKU Names in Usage Table for Job Cost Calculation

mnorland — Tue, 22 Apr 2025 23:46:11 GMT

This could happen if there was at least one run of the job when a task was set to serverless. Alas, the toggle to enable serverless is no more.

Re: Unexpected SKU Names in Usage Table for Job Cost Calculation

lingareddy_Alva — Wed, 23 Apr 2025 04:06:07 GMT

HI @vziog

When you run a job on a shared cluster, you're seeing PREMIUM_JOBS_SERVERLESS_COMPUTE_EU_WEST in the usage table because the system is tracking the serverless resources used to manage and coordinate your job, even when the actual compute happens on a shared cluster. This behavior occurs because:

All Databricks jobs have an orchestration layer that runs on serverless compute
This orchestration layer exists even when your job executes on a shared all-purpose cluster
The job orchestration costs are tracked separately from the actual compute costs

For your second scenario, when running on Job Compute clusters, you see both SKUs because:

PREMIUM_JOBS_COMPUTE represents the actual cluster compute resources
PREMIUM_JOBS_SERVERLESS_COMPUTE_EU_WEST represents the orchestration layer

The NULL values for job_id and job_run_id in some records likely happen because those costs aren't directly tied to a specific job execution but rather to the overall orchestration system.

This dual-charging model ensures that both the orchestration overhead and the actual compute resources are properly accounted for. The serverless component is typically small compared to the cluster compute costs but is still necessary for job coordination.

To accurately calculate job costs, you should include both SKU types for a complete picture of the resources used.

Re: Unexpected SKU Names in Usage Table for Job Cost Calculation

vziog — Wed, 23 Apr 2025 10:27:25 GMT

Thank you all for your replies. @lingareddy_Alva what about @Walter_C and @mnorland mentioned about enabling serverless tasks. Is this possible and how?

Re: Unexpected SKU Names in Usage Table for Job Cost Calculation

lingareddy_Alva — Wed, 23 Apr 2025 13:41:43 GMT

@vziog

They're correct to be concerned about potential serverless task configuration.
The toggle to enable serverless is no more. Databricks has been transitioning away from the explicit serverless toggle in the UI