topic Re: How do I figure out the cost breakdown for Databricks in Data Engineering

How do I figure out the cost breakdown for Databricks

278875 — Sat, 03 Sep 2022 14:04:56 GMT

I'm trying to figure out the cost breakdown for the Databricks usage for my team.

When I go into the Databricks administration console and click Usage when I select to show the usage By SKU it just displays the type of cluster but not the name of it.

When I look at Cost Explorer in AWS and group by Service it shows Databricks Lakehouse Platform which is the majority of the daily cost, when I split by Instance Type the majority of cost is No Instance Type, group by Tag-Cluster Name it shows No Tagkey: ClusterName and by Usage Type it displays MP:dbConsumptionUnit-Units so I'm having a hard time determining what is being used under Databricks.

Any help would be appreciated, thank you

Re: How do I figure out the cost breakdown for Databricks

Prabakar — Tue, 06 Sep 2022 04:33:10 GMT

Hi @Dotan Schachter You can check this notebook.

https://docs.databricks.com/_static/notebooks/usage-delivery-analysis-dashboard.html

Here you have the option to select the clusterName/clusterId that can help you to understand the usage of each cluster or if you have customTag for each team then you can use it to calculate the billing for that team.

You can refer to the usage analysis doc for more details.

https://docs.databricks.com/administration-guide/account-settings/usage-analysis.html

Re: How do I figure out the cost breakdown for Databricks

Vidula — Wed, 21 Sep 2022 05:03:19 GMT

Hi @Dotan Schachter

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: How do I figure out the cost breakdown for Databricks

stefnhuy — Fri, 06 Oct 2023 12:18:24 GMT

Hey there,

I can totally relate to your struggle with deciphering the cost breakdown for Databricks usage—it can be a bit of a maze, can't it? But fear not, I've been down that rabbit hole myself, and I might just have the solution you're looking for.

First off, it's essential to ensure that you have proper tagging set up for your Databricks resources. It seems like you're encountering issues because of missing tags. In AWS, tagging is your best friend for cost allocation. Ensure that each Databricks cluster and resource is appropriately tagged with a ClusterName or any other identifier that makes sense for your team.

Once you have your tags in place, revisit the Databricks administration console. By filtering or grouping your usage by tags, you should be able to get a more granular breakdown of your costs. You can then correlate this with the information you see in AWS Cost Explorer.

I know it sounds a bit complex, but trust me, proper tagging is the linchpin to a clearer cost breakdown. Additionally, if you're interested in exploring how successful testing projects can impact cost optimization further, you might want to check out "Andersen Financial Services: Explore How It Elevates By Successful Testing Project." They've shared some valuable insights on this topic that could complement your efforts.

Re: How do I figure out the cost breakdown for Databricks

MuthuLakshmi — Mon, 06 Nov 2023 05:43:13 GMT

Please check the below docs for usage related informations.

The Billable Usage Logs:

https://docs.databricks.com/en/administration-guide/account-settings/usage.html

You can filter them using tags for more precise information which you are looking for:

https://docs.databricks.com/en/administration-guide/account-settings/usage-detail-tags.html

And for analysing specifically, what exactly is the utilization on a cluster, Cluster Metrics are used generally https://docs.databricks.com/en/compute/cluster-metrics.html