topic Re: When the billing time starts for the cluster? in Data Engineering

When the billing time starts for the cluster?

Mariusz_Cyp — Sat, 11 Dec 2021 09:18:38 GMT

Hi All,

I'm just wondering when exactly the billing time starts for the DataBricks cluster? Is starting time included? If cluster creation time takes 3 minutes and query execution only 2, will I pay for 2 or 5?

Thanks in advance!

Re: When the billing time starts for the cluster?

Hubert-Dudek — Sat, 11 Dec 2021 10:23:17 GMT

Officialy it is for compute time so DBU will be per minute since you start cluster (so 5 in your example). Additionally you will pay for at least 5 minute for instance in your cloud provider (from my experience usually it is a bit more as terminating and cleaning resources takes time). In databricks roadmap there is serverless offering so in future it will be a bit less complicated and more cost effective.

Re: When the billing time starts for the cluster?

-werners- — Mon, 13 Dec 2021 08:15:15 GMT

Correct. The moment your cloud providers starts provisioning and stops provisioning is the time interval you are billed.

So for short jobs that is a lot of overhead.

What I do is to use a pool, in which I try to schedule as many jobs as possible, like that you have a lot less startup time.

Re: When the billing time starts for the cluster?

franco_patano — Mon, 13 Dec 2021 18:59:45 GMT

Billing for databricks DBUs starts when Spark Context becomes available. Billing for the cloud provider starts when the request for compute is received and the VMs are starting up.