topic Attempt to use a custom container with an instance pool fails in Data Engineering

Attempt to use a custom container with an instance pool fails

mrstevegross — Tue, 25 Mar 2025 14:50:33 GMT

I am trying to run a job with (1) custom containers, and (2) via an instance pool. Here's the setup:

The custom container is just the DBR-provided `databricksruntime/standard:12.2-LTS`
The instance pool is defined via the UI (see screenshot, below).
At runtime, the error I get is: `The target instance pool InstancePoolId(<redacted>) does not have docker images configured, thus not supporting cluster creation with docker image. Please update your cluster attribute or create a separate instance pool for docker image clusters.`

I do not see any setting on the instance pool related to enabling docker images, however. I found one related post on the forum, but is a slightly different scenario and doesn't have a clear resolution.

Any pointers?

Thanks,

--Steve

Re: Attempt to use a custom container with an instance pool fails

Isi — Wed, 26 Mar 2025 12:36:17 GMT

Hey Steve @mrstevegross ,

To preload container services in a pool, you must do it via the Databricks API, since this option is not available through the UI.

That said, it’s important to keep in mind that pools are just compute infrastructure, not clusters themselves. Pools don’t define runtime behavior, that’s something you set when you create a cluster that uses the pool.

Hope this helps 🙂

Isi

Re: Attempt to use a custom container with an instance pool fails

mrstevegross — Wed, 26 Mar 2025 13:46:06 GMT

>To preload container services in a pool, you must do it via the Databricks API, since this option is not available through the UI.

I'm not trying to "preload" it, I just want to my cluster to (1) use a container, and (2) use my pool. I'm aware that the create instance pool API offers a "preloaded_images" mechanism, but there's nothing in the docs that says "preloaded_images" are *required* if a cluster create request uses the docker_image option. Can you clarify?

Re: Attempt to use a custom container with an instance pool fails

mrstevegross — Wed, 26 Mar 2025 17:39:40 GMT

I think I have solved this. I added a URL for `preloaded_docker_image` to my instance pool, and the job worked correctly.

This suggests that the DBR docs for preloaded_docker_image are incomplete; they should clarify that a user must add an entry in order to use the docker image feature.

Re: Attempt to use a custom container with an instance pool fails

mrstevegross — Mon, 07 Apr 2025 14:33:56 GMT

If anyone from DBR is monitoring this thread, can y'all confirm my understanding and--if so--update the docs to reflect this requirement?

Re: Attempt to use a custom container with an instance pool fails

Advika — Mon, 07 Apr 2025 15:34:53 GMT

Hello @mrstevegross!

Thanks for sharing your findings. While the docs mention this setting, they don’t clearly state it’s required. Your observation makes sense and clarifying this would help others avoid confusion. I’ll share this with the internal teams.