topic Re: Monitoring pool usage in Administration & Architecture

Monitoring pool usage

nayan_wylde — Tue, 01 Jul 2025 18:51:43 GMT

Hi All,

I am working on creating a dashboard for databricks instance pools. I am capturing maximum usage and scheduled a jpb to capture the info every 15 mins and I aggregate to see if any point the max usage is greater then 85% of capacity, Is there any other metrices that we can capture. Ideas are appreciated

Re: Monitoring pool usage

szymon_dybczak — Tue, 01 Jul 2025 19:21:11 GMT

Hi @nayan_wylde,

Below articles contain some useful metrics

https://community.databricks.com/t5/technical-blog/queries-for-cost-attribution-using-system-tables/ba-p/76558

https://medium.com/@wesbert/databricks-system-tables-overview-the-node-types-and-clusters-tables-b5f17082c8b9

Re: Monitoring pool usage

nayan_wylde — Tue, 01 Jul 2025 20:45:00 GMT

I am looking to build some custom metrices for pools. Some of them are available in dbdemos python package that refers system tables. I am looking to forecast my pool usage for next 30 days I am using ai_forecast function for it. Is there any other custom metrices we can capture for pools

Re: Monitoring pool usage

nayan_wylde — Wed, 02 Jul 2025 18:02:03 GMT

The ai_forecast function did the magic and it have the forecast on pool usage that predicted for next 30 days