Monitoring pool usage

[nayan_wylde]

Hi All,

I am working on creating a dashboard for databricks instance pools. I am capturing maximum usage and scheduled a jpb to capture the info every 15 mins and I aggregate to see if any point the max usage is greater then 85% of capacity, Is there any other metrices that we can capture. Ideas are appreciated

[szymon_dybczak]

Hi @nayan_wylde,

Below articles contain some useful metrics

https://community.databricks.com/t5/technical-blog/queries-for-cost-attribution-using-system-tables/...

https://medium.com/@wesbert/databricks-system-tables-overview-the-node-types-and-clusters-tables-b5f...

[nayan_wylde · Accepted solution]

I am looking to build some custom metrices for pools. Some of them are available in dbdemos python package that refers system tables. I am looking to forecast my pool usage for next 30 days I am using ai_forecast function for it. Is there any other custom metrices we can capture for pools

[nayan_wylde]

The ai_forecast function did the magic and it have the forecast on pool usage that predicted for next 30 days