cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Monitoring VM costs using cluster pools

radothede
New Contributor III

Hello,

With ref to docs https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage-detail-tags cluster tags are not propagated to VM when created within a pool.

Is there any workaround for monitoring VM costs using cluster pools (job clusters)?

Sharing Your approach and experiences most appreciated.

 

3 REPLIES 3

radothede
New Contributor III

Thanks @Retired_mod for Your reply!

radothede
New Contributor III

Dear @Retired_mod ,

as You mentioned, Databricks does not provide out of the box support for VM usage monitoring for job clusters created from cluster pool.

If we really want to use cluster pool, I would consider:

1) splitting the pool into separate pools on the granularity level that reflects our needs for costs reporting,

2) extrapolate the DBU costs to VM usage costs (as they are very similar) on the job level,

3) tune pools (setting min idle, autoterminate) to minimalize idle time.

An alternative approach would be to use job computes - not from the cluster pool, and then use tagging.

 

-werners-
Esteemed Contributor III

I am leaving pools more and more and using reusable job clusters more and more.  Like that you have more control.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group