topic Pre-loading docker images to cluster pool instances still requires docker URL at cluster creation in Administration & Architecture

Pre-loading docker images to cluster pool instances still requires docker URL at cluster creation

NadithK — Wed, 23 Oct 2024 03:24:21 GMT

I am trying to pre-load a docker image to a Databricks cluster pool instance.
As per this article I used the REST API to create the cluster pool and defined a custom Azure container registry as the source for the docker images.
https://learn.microsoft.com/en-us/azure/databricks/compute/pools#dcs

Pool gets created successfully.

But now whenever I try to create a cluster using that pool, Databricks still asks me to define a docker source in the cluster creation step. Gives me the error

"'docker image' must be provided for cluster created with instance pool xxxxxxxxx"

Does this mean that I still have to define the same docker when I create the cluster. If so, what is the point of defining it in the cluster pool. I want to define the custom docker container at the cluster pool level so users who use the pool to create their clusters don't have to know about the underlying docker image used or the credentials to access the docker source.

Can anyone please help me clear this up.

Re: Pre-loading docker images to cluster pool instances still requires docker URL at cluster creatio

krupakar1329 — Thu, 25 Sep 2025 11:14:46 GMT

Any updates on this error?

Re: Pre-loading docker images to cluster pool instances still requires docker URL at cluster creatio

krupakar1329 — Fri, 26 Sep 2025 09:20:50 GMT

@NadithK Pre-loading Docker images to cluster pool instances is for performance optimization (faster cluster startup), but you still must specify the Docker image in your cluster configuration. The pre-loading doesn't eliminate the requirement to declare which image the cluster should use.

This design allows you to have multiple clusters using different Docker images from the same pool while benefiting from the pre-loaded images for faster initialization.