Attempt to use a custom container with an instance pool fails

[mrstevegross]

I am trying to run a job with (1) custom containers, and (2) via an instance pool. Here's the setup:

1. The custom container is just the DBR-provided `databricksruntime/standard:12.2-LTS`
2. The instance pool is defined via the UI (see screenshot, below).
3. At runtime, the error I get is: `The target instance pool InstancePoolId(<redacted>) does not have docker images configured, thus not supporting cluster creation with docker image. Please update your cluster attribute or create a separate instance pool for docker image clusters.`

I do not see any setting on the instance pool related to enabling docker images, however. I found one related post on the forum, but is a slightly different scenario and doesn't have a clear resolution.

Any pointers?

Thanks,

--Steve

[Isi]

Hey Steve @mrstevegross ,

To preload container services in a pool, you must do it via the Databricks API, since this option is not available through the UI.

That said, it’s important to keep in mind that pools are just compute infrastructure, not clusters themselves. Pools don’t define runtime behavior, that’s something you set when you create a cluster that uses the pool.

Hope this helps 🙂

Isi

[mrstevegross]

>To preload container services in a pool, you must do it via the Databricks API, since this option is not available through the UI.

I'm not trying to "preload" it, I just want to my cluster to (1) use a container, and (2) use my pool. I'm aware that the create instance pool API offers a "preloaded_images" mechanism, but there's nothing in the docs that says "preloaded_images" are *required* if a cluster create request uses the docker_image option. Can you clarify?

[mrstevegross]

I think I have solved this. I added a URL for `preloaded_docker_image` to my instance pool, and the job worked correctly.

This suggests that the DBR docs for preloaded_docker_image are incomplete; they should clarify that a user must add an entry in order to use the docker image feature.

[mrstevegross]

If anyone from DBR is monitoring this thread, can y'all confirm my understanding and--if so--update the docs to reflect this requirement?

[Advika · Accepted solution]

Hello @mrstevegross!

Thanks for sharing your findings. While the docs mention this setting, they don’t clearly state it’s required. Your observation makes sense and clarifying this would help others avoid confusion. I’ll share this with the internal teams.