topic Possible to programmatically adjust Databricks instance pool more intelligently? in Administration & Architecture

Possible to programmatically adjust Databricks instance pool more intelligently?

mrstevegross — Mon, 17 Mar 2025 19:56:44 GMT

We'd like to adopt Databricks instance pool in order to reduce instance-acquisition times (a significant contributor to our test latency). Based on my understanding of the docs, the main levers we can control are: min instance count, max instance count, and idle termination time.

However, our usage pattern is bursty: every 2 hours we run a full test suite, which means ~100 instances need to be acquired. After the test run is done, we release the instances (until the next ~2 hour run). Are there any options for managing the instance pool to better take advantage of the predictable bursty usage pattern?

Re: Possible to programmatically adjust Databricks instance pool more intelligently?

Isi — Mon, 17 Mar 2025 20:34:41 GMT

Hi @mrstevegross ,

I’d like to clarify how those ~100 instances are used:

Are all 100 instances required simultaneously (fully parallel execution), or do they ramp up progressively based on test needs?

How long do the tests typically run before the instances are released?

If all 100 instances are needed at once (fully parallel execution):

• Pre-warm the pool: Around 1h 55m before each test run, trigger 100 API calls to acquire instances ahead of time.
•Set idle_termination = 0 to ensure instances shut down as soon as tests complete.

•This guarantees minimal latency for instance allocation.

If instance usage is progressive (some start earlier, some later):

• Adjust idle_termination_time so instances stay warm long enough to complete dependent tasks, but don’t stay idle too long.
•Set a small min instance count to keep a few warm if the progressive ramp-up is predictable.

If you can share more about how your test execution scales over time, I’d be happy to refine the strategy further!

🙂

Isi

Re: Possible to programmatically adjust Databricks instance pool more intelligently?

mrstevegross — Mon, 17 Mar 2025 21:15:09 GMT

>If all 100 instances are needed at once (fully parallel execution):

>• Pre-warm the pool: Around 1h 55m before each test run, trigger 100 API calls to acquire instances ahead of time.
>•Set idle_termination = 0 to ensure instances shut down as soon as tests complete.

Which API call do you have in mind to "acquire" instances? (Looking at the API docs it's not obvious which call you have in mind)

Thanks,

--Steve

Re: Possible to programmatically adjust Databricks instance pool more intelligently?

Isi — Mon, 17 Mar 2025 22:08:33 GMT

Hi Steve,

If the goal is to pre-warm 100 instances in the Databricks Instance Pool, you could create a temporary job that will request instances from the pool. This ensures that Databricks provisions the required instances before the actual test run.

The cluster will force Databricks to allocate 100 instances from the pool. Since autotermination_minutes can be set to 10 (for example), it will shut down shortly after the instances are ready. Your actual test jobs will then be able to use the pre-warmed instances instantly.

I think is and idea that could work and pre warm instances

🙂

Isi

Re: Possible to programmatically adjust Databricks instance pool more intelligently?

mrstevegross — Tue, 18 Mar 2025 00:49:51 GMT

If the goal is to pre-warm 100 instances in the Databricks Instance Pool, you could create a temporary job that will request instances from the pool. This ensures that Databricks provisions the required instances before the actual test run.
The cluster will force Databricks to allocate 100 instances from the pool. Since autotermination_minutes can be set to 10 (for example), it will shut down shortly after the instances are ready. Your actual test jobs will then be able to use the pre-warmed instances instantly.

Yeah, that makes sense, and could be a workable tactic. Thanks!