cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Monitoring VM costs using cluster pools

radothede
New Contributor III

Hello,

With ref to docs https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage-detail-tags cluster tags are not propagated to VM when created within a pool.

Is there any workaround for monitoring VM costs using cluster pools (job clusters)?

Sharing Your approach and experiences most appreciated.

 

5 REPLIES 5

Kaniz
Community Manager
Community Manager

Hi @radothede, You’re correct that in Databricks, cluster tags are not propagated to VMs when created within a pool. This can make it challenging to monitor VM costs using cluster pools, especially for job clusters.

While there isn’t a direct workaround to propagate these tags to VMs within a pool, you can still monitor and manage costs effectively by using a combination of strategies:

  1. DBU Usage Reports: The Databricks Unit (DBU) is the underlying unit of consumption within the platform. By tagging clusters and pools, you can monitor cost and accurately attribute Databricks usage to your organization’s business units and teams. These tags propagate to detailed DBU usage reports.

  2. Cluster Policies: Since costs are calculated through the usage of compute resources, and more specif...4. Different attributes of cluster policies can restrict DBU consumption and effectively manage costs o...4.

  3. Cost Management Tools: Use Databricks Admin tools to find the balance and control costs without thro...productivity.

 

I hope this helps! If you have any more questions, feel free to ask.

 

radothede
New Contributor III

Thanks @Kaniz for Your reply!

Kaniz
Community Manager
Community Manager

Hi @radothede,

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.

If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!

radothede
New Contributor III

Dear @Kaniz ,

as You mentioned, Databricks does not provide out of the box support for VM usage monitoring for job clusters created from cluster pool.

If we really want to use cluster pool, I would consider:

1) splitting the pool into separate pools on the granularity level that reflects our needs for costs reporting,

2) extrapolate the DBU costs to VM usage costs (as they are very similar) on the job level,

3) tune pools (setting min idle, autoterminate) to minimalize idle time.

An alternative approach would be to use job computes - not from the cluster pool, and then use tagging.

 

-werners-
Esteemed Contributor III

I am leaving pools more and more and using reusable job clusters more and more.  Like that you have more control.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!