topic When does the cost of JOB COMPUTE start to be calculated? in Get Started Discussions

When does the cost of JOB COMPUTE start to be calculated?

himanmon — Fri, 05 Jul 2024 06:27:37 GMT

I'm trying to run a workflow with job compute.
Job compute needs to be pending for about 5 to 7 minutes before executing the workflow. I think it takes time to find a suitable instance in the cloud, configure the environment, install libraries, etc.

And Databricks charges us by the time we use it. So, is the 5-7 minutes required for pending included in the cost? Or is only the time when pending ends and the workflow actually starts included in the cost?

Thank you for your answer.

Re: When does the cost of JOB COMPUTE start to be calculated?

Walter_C — Sat, 06 Jul 2024 15:27:16 GMT

The cost of JOB COMPUTE starts to be calculated from the moment the job starts running.
You can monitor the cost of jobs that use serverless compute for workflows by querying the billable usage system table.

Re: When does the cost of JOB COMPUTE start to be calculated?

himanmon — Tue, 09 Jul 2024 05:00:24 GMT

Thank you for your answer. That means pending time is also included in the cost, right?

Re: When does the cost of JOB COMPUTE start to be calculated?

jacovangelder — Tue, 09 Jul 2024 05:45:08 GMT

There is a big discrepancy to take note of here. It is the difference between Databricks job costs (DBUs) and billing for the cloud provider (the actual VM doing the the work).

Billing for the Databricks job starts when the Spark context is being initiated (you can see so in the event log). It is usually at 4-5 minutes in.
Billing for the cloud provider starts the moment you press the start button on your pipeline (this will provision the VM in 5 minutes).

Hope this helps!