cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How do I figure out the cost breakdown for Databricks

278875
New Contributor

I'm trying to figure out the cost breakdown for the Databricks usage for my team.

When I go into the Databricks administration console and click Usage when I select to show the usage By SKU it just displays the type of cluster but not the name of it.

When I look at Cost Explorer in AWS and group by Service it shows Databricks Lakehouse Platform which is the majority of the daily cost, when I split by Instance Type the majority of cost is No Instance Type, group by Tag-Cluster Name it shows No Tagkey: ClusterName and by Usage Type it displays MP:dbConsumptionUnit-Units so I'm having a hard time determining what is being used under Databricks.

Any help would be appreciated, thank you

4 REPLIES 4

Prabakar
Esteemed Contributor III

Hi @Dotan Schachterโ€‹ You can check this notebook.

https://docs.databricks.com/_static/notebooks/usage-delivery-analysis-dashboard.html

imageHere you have the option to select the clusterName/clusterId that can help you to understand the usage of each cluster or if you have customTag for each team then you can use it to calculate the billing for that team.

You can refer to the usage analysis doc for more details.

https://docs.databricks.com/administration-guide/account-settings/usage-analysis.html

Vidula
Honored Contributor

Hi @Dotan Schachterโ€‹ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

stefnhuy
New Contributor III

Hey there,

I can totally relate to your struggle with deciphering the cost breakdown for Databricks usageโ€”it can be a bit of a maze, can't it? But fear not, I've been down that rabbit hole myself, and I might just have the solution you're looking for.

First off, it's essential to ensure that you have proper tagging set up for your Databricks resources. It seems like you're encountering issues because of missing tags. In AWS, tagging is your best friend for cost allocation. Ensure that each Databricks cluster and resource is appropriately tagged with a ClusterName or any other identifier that makes sense for your team.

Once you have your tags in place, revisit the Databricks administration console. By filtering or grouping your usage by tags, you should be able to get a more granular breakdown of your costs. You can then correlate this with the information you see in AWS Cost Explorer.

I know it sounds a bit complex, but trust me, proper tagging is the linchpin to a clearer cost breakdown. Additionally, if you're interested in exploring how successful testing projects can impact cost optimization further, you might want to check out "Andersen Financial Services: Explore How It Elevates By Successful Testing Project." They've shared some valuable insights on this topic that could complement your efforts.

MuthuLakshmi
New Contributor III

Please check the below docs for usage related informations.

The Billable Usage Logs:

https://docs.databricks.com/en/administration-guide/account-settings/usage.html

You can filter them using tags for more precise information which you are looking for:

https://docs.databricks.com/en/administration-guide/account-settings/usage-detail-tags.html

And for analysing specifically, what exactly is the utilization on a cluster, Cluster Metrics are used generally https://docs.databricks.com/en/compute/cluster-metrics.html

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group