cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Pre-loading docker images to cluster pool instances still requires docker URL at cluster creation

NadithK
Contributor

I am trying to pre-load a docker image to a Databricks cluster pool instance.
As per this article I used the REST API to create the cluster pool and defined a custom Azure container registry as the source for the docker images.
https://learn.microsoft.com/en-us/azure/databricks/compute/pools#dcs 

Pool gets created successfully.

But now whenever I try to create a cluster using that pool, Databricks still asks me to define a docker source in the cluster creation step. Gives me the error

"'docker image' must be provided for cluster created with instance pool xxxxxxxxx"

Does this mean that I still have to define the same docker when I create the cluster. If so, what is the point of defining it in the cluster pool. I want to define the custom docker container at the cluster pool level so users who use the pool to create their clusters don't have to know about the underlying docker image used or the credentials to access the docker source.

Can anyone please help me clear this up.

2 REPLIES 2

krupakar1329
New Contributor II

Any updates on this error?

krupakar1329
New Contributor II

@NadithK Pre-loading Docker images to cluster pool instances is for performance optimization (faster cluster startup), but you still must specify the Docker image in your cluster configuration. The pre-loading doesn't eliminate the requirement to declare which image the cluster should use.

This design allows you to have multiple clusters using different Docker images from the same pool while benefiting from the pre-loaded images for faster initialization.

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now