How do I compare cost between databricks gcp and azure databricks ?

[Tahseen0354]

I have a databricks job running in azure databricks. A similar job is also running in databricks gcp. I would like to compare the cost. If I assign a custom tag to the job cluster running in azure databricks, I can see the cost incurred by that job in azure cost analysis dashboard. How do I see the same in GCP ? What is a good way to do something like that ?

[Abishek]

Hello @Md Tahseen Anam​ 

For GCP Can you please check the below links for usage deatils :

View billable usage using the account console

https://docs.gcp.databricks.com/administration-guide/account-settings-gcp/usage.html

Analyze billable usage log data :

https://docs.gcp.databricks.com/administration-guide/account-settings/usage-analysis.html

 https://www.databricks.com/product/gcp-pricing/instance-types

[Tahseen0354]

Thanks! I think the following link can be very useful:

Analyze billable usage log data :

https://docs.gcp.databricks.com/administration-guide/account-settings/usage-analysis.html

Is there a similar way in azure to analyze the usage log data from the usage log file ? Can you share any similar link for azure ?

[Abishek]

For Azure Databricks check the below links :

https://www.databricks.com/product/azure-pricing

https://azure.microsoft.com/en-us/pricing/details/databricks/

https://azure.microsoft.com/en-us/pricing/details/databricks/#pricing

https://www.azure.cn/en-us/pricing/details/databricks/

[Tahseen0354]

Thank you for your reply. The links for azure shows the pricing details. I was looking for a way to analyze the cost associated with jobs and clusters both in azure and gcp so that we can easily compare them. Seems like the way we analyze cost in gcp is different to how we do it in azure. Would be nice to know if we can use a cost log file to create a notebook dashboard to analyze the cost in azure databricks.

[Dicer]

For Azure., you can go to cost management to check your expense.

[Tahseen0354]

Thank you for your reply.

[Tahseen0354]

I do not have any other solution other than those mentioned above. I have already explored them. But I was actually looking for something like an auto generated report to see the cost incurred by each individual job, user and cluster in each workspace for a period of time. Imagine there is a new user who has just started using databricks. How will that user know how much cost he/she is incurring by his/her each activity in the workspace from the very beginning without any help from some experienced user?

[Dicer]

@Md Tahseen Anam​ I haven't used GCP, but there is a Cost Management tool in GCP

https://cloud.google.com/blog/topics/cost-management/cost-management-tools-in-google-cloud-console

[Hubert-Dudek]

Additionally, it is two completely different systems, and most importantly, in Azure, databricks are natively integrated, so in Azure, you take everything from cost management.

DBU is similar in cost and can be easy compared. Regarding all other expenses (VMs, storage) you need to take from cost reports.

[Anonymous]

Hi @Md Tahseen Anam​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

[Anonymous]

Hello,

Databricks on Google Cloud

Open. Built on open standards, open APIs and open infrastructure so you can access, process and analyze data on your terms.

Optimized. Deploy Data bricks on Google Kubernetes Engine, the first Kubernetes-based Data bricks runtime on any cloud, to get insights faster.

Integrated.

any one can post regarding it .

[Felix]

Hi @Md Tahseen Anam​ , have you had a look at https://databrickslabs.github.io/overwatch/gettingstarted/ before? As of now it's only available on AWS and Azure, but I think this is the kind of granularity you're looking for. In general, you need to factor in the costs for DBU and cloud provider compute - unless you want to go really deep and consider secondary costs like storage, v-nets, ... as well. In your specific case I would recommend to give each user their respective cluster (or personal compute to keep costs low) and tag these respectively. You can then aggregate over the tags via the standard cost monitoring solutions on Azure and GCP. Once the serverless offering on the Databricks platform includes workflows and general compute as well, this will provide you with additional insight and opportunity to control cost.

[Own]

In Azure, you can use Cost Management to track your expenses incurred by Databricks instance.