topic Re: Timeout on docker pull in Databricks Container Services in Administration & Architecture

Timeout on docker pull in Databricks Container Services

kch — Fri, 06 Dec 2024 10:26:10 GMT

Hello,

There is a timeout that limits the size of images used in Docker Container Service. When using images containing large ML libraries, the size often exceeds the limit that could be pulled. Is there any plan to add parametrization of this timeout? Or at least increase it?

Re: Timeout on docker pull in Databricks Container Services

Walter_C — Fri, 06 Dec 2024 12:21:10 GMT

Currently, the timeout for pulling Docker images in Databricks Container Service is fixed at 10 minutes and cannot be changed.

The primary recommendation to address this issue is to reduce the size of your Docker image. This can be achieved by flattening the Docker image to minimize the number of layers, as each layer must be written to disk sequentially, which can create a bottleneck.

Re: Timeout on docker pull in Databricks Container Services

kch — Mon, 09 Dec 2024 08:37:41 GMT

@Walter_C Thank you for your response. Unfortunately, this doesn't resolve my issue. Using the GPU version of Torch even without additional dependencies comes very close to the limit. Once I add the necessary components for training or using the ML model, it’s easy to exceed the limit, even with everything computed in a single layer, caches cleared, and dependencies optimized. Considering how common this situation has become, I’m curious if there are any plans to parameterize or adjust the limit?

Re: Timeout on docker pull in Databricks Container Services

kch — Thu, 13 Mar 2025 05:45:11 GMT

Are there any new or planned changes in the policy?