topic Re: Cost of individual jobs running on a shared Databricks cluster in Data Engineering

Cost of individual jobs running on a shared Databricks cluster

Ashish — Mon, 10 Jan 2022 08:46:47 GMT

Hi All,

I am working on a requirement where I need to calculate the cost of each spark job individually on a shared Azure/AWS Databricks cluster. There can be multiple jobs running on the cluster parallelly.

Cost needs to be calculated after job completion and it has to be calculated programatically

So, I'm looking for an API

-which can fetch the cost of each individual job directly. It would be an ideal solution.

-which can give resource usage of a job in terms of vcore-seconds in Databricks cluster as available in Yarn through Resource Manager API

In a Yarn cluster, I'm using the following approach to calculate the cost of each spark application individually:

Cost = (ec2 instance cost per hour per core * application's vcore-hour) + EMR Fee

In Yarn, resource manager provides resource usage of each spark application in terms of memory-seconds and vcore-seconds.

Re: Cost of individual jobs running on a shared Databricks cluster

-werners- — Mon, 10 Jan 2022 09:17:22 GMT

Do you have to calculate this beforehand? Because that is pretty hard to predict.

Especially if you use autoscaling.

Besides hardware provisioning cost, you also have the DBUs.

And the type of VMs which is used, job cluster or interactive cluster etc.

Now, on Azure there is an excellent cost breakdown possible, I suppose on AWS this is also possible.

But that is of course post hoc, so your jobs have already run.

Re: Cost of individual jobs running on a shared Databricks cluster

alexott — Mon, 10 Jan 2022 10:14:36 GMT

There is a built-in functionality for getting the costs:

AWS - https://docs.databricks.com/administration-guide/account-settings-e2/usage.html
Azure - via built-in Cost Management + Billing

The main problem with that functionality is that the smallest granularity you get is cluster/job because it relies on the cluster/job tags. And also there are problems with compute costs when nodes are obtained from the instance pools, as the nodes aren't re-tagged when used.

These problems could be solved by using the project Overwatch that allows to get more granular data, like costs per notebook/user/...

Re: Cost of individual jobs running on a shared Databricks cluster

Ashish — Wed, 12 Jan 2022 04:10:24 GMT

Cost needs to be calculated after job completion and it has to be calculated programatically (which I missed to mention).

And I'm calculating the cost as follow:

Cost = (ec2 instance cost per hour per core * job's vcore-hour) + EMR Fee

Therefore, I need an API which can give resource usage of the job in terms of vcore-seconds/hours in Databricks cluster as available in Yarn through Resource Manager API.

Is there any API available ? or is there any other way to calculate the cost ?

And if there is an API which can give the cost of each job directly, that would be great.

Re: Cost of individual jobs running on a shared Databricks cluster

Ashish — Sat, 15 Jan 2022 09:56:18 GMT

No, its not answered yet.