topic Re: preloaded_docker_images: how do they work? in Get Started Discussions

preloaded_docker_images: how do they work?

mrstevegross — Mon, 13 Jan 2025 16:44:15 GMT

At my org, when we start a databricks cluster, it oftens takes awhile to become available (due to (1) instance provisioning, (2) library loading, and (3) init script execution). I'm exploring whether an instance pool could be a viable strategy for improving cluster execution time.

I see there's a feature called "preloaded_docker_images" (https://docs.databricks.com/api/workspace/instancepools/get#preloaded_docker_images), but the docs are limited. Is there canonincal documentation the explains:

When are the docker images loaded in the lifecycle of the instance?
Given that you can supply N images, how does container isolation work? I only need to load one container, but it's unclear to me if my Spark job will run "inside" that container? (Given that I'm using the container to preload jars, I'm pretty sure the Spark job needs to be able to see those jars!)

Re: preloaded_docker_images: how do they work?

Alberto_Umana — Mon, 13 Jan 2025 16:59:09 GMT

Hi @mrstevegross,

About your cluster startup time, how long does it take to come up?

1 - Docker images specified in the preloaded_docker_images field are loaded when the instance pool is created or when instances are added to the pool. This means that the images are pulled and cached on the instances before they are used for running jobs.

When you specify a Docker image for your Databricks cluster, the entire cluster runs within that Docker container. This means that all Spark jobs executed on the cluster will run inside the specified Docker container.

2. Since your Spark job runs inside the Docker container, it will have access to any jars or libraries that are preloaded within that container. This ensures that your Spark job can see and use the preloaded jars as expected.

Please be aware of some limitations: https://docs.databricks.com/en/compute/custom-containers.html

Re: preloaded_docker_images: how do they work?

mrstevegross — Mon, 13 Jan 2025 17:02:20 GMT

>When you specify a Docker image for your Databricks cluster, the entire cluster runs within that Docker container.

Just to clarify: are you saying that the Databricks job request itself says which container to use?

>Please be aware of some limitations: https://docs.databricks.com/en/compute/custom-containers.html

Roger that, reading docs now.

Thanks!

Re: preloaded_docker_images: how do they work?

mrstevegross — Mon, 13 Jan 2025 17:25:18 GMT

>Just to clarify: are you saying that the Databricks job request itself says which container to use?

I see here (https://docs.databricks.com/api/workspace/clusters/create#docker_image) that the create-cluster request can include an image-to-load. How does that interact with the instance pool's "preloaded_docker_images" feature?

Re: preloaded_docker_images: how do they work?

Alberto_Umana — Mon, 13 Jan 2025 19:39:43 GMT

Hi @mrstevegross, not exactly it should come with the API request.

When you create a cluster using an instance pool with preloaded Docker images, the cluster can use one of the preloaded images if it matches the docker_image specified in the create-cluster request. If the specified docker_image is not preloaded in the instance pool, the cluster will load the specified image, which may take additional time.

Re: preloaded_docker_images: how do they work?

mrstevegross — Mon, 13 Jan 2025 19:48:14 GMT

> if it matches the docker_image specified in the create-cluster request.

Aha, good to know. Can y'all update the reference docs to clarify these semantics?

Re: preloaded_docker_images: how do they work?

Alberto_Umana — Mon, 13 Jan 2025 20:48:20 GMT

Sure, I will inform the team in charge of it to review it.

Re: preloaded_docker_images: how do they work?

naha3456 — Tue, 29 Apr 2025 12:15:15 GMT

Hello, when we specify docker image with credentials in instance pool configuration, should we also specify credentials in cluster configuration?. as we already have image pulled into the pool instance.