cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Instance Pool Usage

muir
New Contributor II

We have instance pools setup with a maximum capacity and are looking at ways to monitor the usage to help with our capacity planning.

I have been using the system tables to track how many nodes are being used within a pool at a point in time but it appears to be that the node_timeline table only tracks nodes as they are used and so idle nodes are not counted.

From the docs idle nodes count towards the max capacity of the pool (along with nodes in use) so I cannot see how to capture this from the system tables. Is there any way of doing this?

I realise there is an api that I could hit to get the status of the pool periodically but was hoping there was a query option first.

Thanks

 

1 ACCEPTED SOLUTION

Accepted Solutions

nick533
New Contributor II

The system tables like  only track active node usage, not idle nodes. Since idle nodes also count toward the pool's maximum capacity, their status cannot be directly captured through queries on system tables. To monitor both active and idle nodes within a pool, youโ€™ll need to use the API to periodically fetch the pool status, as it provides comprehensive details, including idle nodes. Unfortunately, there isnโ€™t a direct query-based solution to achieve this.

View solution in original post

3 REPLIES 3

Walter_C
Databricks Employee
Databricks Employee

Unfortunately as of now the API endpoint will be the best approach.

nick533
New Contributor II

The system tables like  only track active node usage, not idle nodes. Since idle nodes also count toward the pool's maximum capacity, their status cannot be directly captured through queries on system tables. To monitor both active and idle nodes within a pool, youโ€™ll need to use the API to periodically fetch the pool status, as it provides comprehensive details, including idle nodes. Unfortunately, there isnโ€™t a direct query-based solution to achieve this.

TuckerGage
New Contributor II

I also using it and it's working properly.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group