topic Re: Instance Pool Usage in Data Engineering

Instance Pool Usage

muir — Thu, 23 Jan 2025 16:36:06 GMT

We have instance pools setup with a maximum capacity and are looking at ways to monitor the usage to help with our capacity planning.

I have been using the system tables to track how many nodes are being used within a pool at a point in time but it appears to be that the node_timeline table only tracks nodes as they are used and so idle nodes are not counted.

From the docs idle nodes count towards the max capacity of the pool (along with nodes in use) so I cannot see how to capture this from the system tables. Is there any way of doing this?

I realise there is an api that I could hit to get the status of the pool periodically but was hoping there was a query option first.

Thanks

Re: Instance Pool Usage

Walter_C — Thu, 23 Jan 2025 19:40:51 GMT

Unfortunately as of now the API endpoint will be the best approach.

Re: Instance Pool Usage

nick533 — Fri, 24 Jan 2025 05:57:35 GMT

The system tables like only track active node usage, not idle nodes. Since idle nodes also count toward the pool's maximum capacity, their status cannot be directly captured through queries on system tables. To monitor both active and idle nodes within a pool, you’ll need to use the API to periodically fetch the pool status, as it provides comprehensive details, including idle nodes. Unfortunately, there isn’t a direct query-based solution to achieve this.

Re: Instance Pool Usage

TuckerGage — Wed, 05 Feb 2025 05:17:10 GMT

I also using it and it's working properly.