topic Re: Monitoring VM costs using cluster pools in Administration & Architecture

Monitoring VM costs using cluster pools

radothede — Mon, 20 May 2024 19:54:16 GMT

Hello,

Is there any workaround for monitoring VM costs using cluster pools (job clusters)?

Sharing Your approach and experiences most appreciated.

radothede — Thu, 23 May 2024 07:36:15 GMT

Thanks @Retired_mod for Your reply!

radothede — Thu, 23 May 2024 09:09:40 GMT

as You mentioned, Databricks does not provide out of the box support for VM usage monitoring for job clusters created from cluster pool.

If we really want to use cluster pool, I would consider:

1) splitting the pool into separate pools on the granularity level that reflects our needs for costs reporting,

2) extrapolate the DBU costs to VM usage costs (as they are very similar) on the job level,

3) tune pools (setting min idle, autoterminate) to minimalize idle time.

An alternative approach would be to use job computes - not from the cluster pool, and then use tagging.

-werners- — Thu, 23 May 2024 09:25:41 GMT

I am leaving pools more and more and using reusable job clusters more and more. Like that you have more control.