topic Re: Failed to start cluster: Large docker image in Administration & Architecture

Failed to start cluster: Large docker image

NateJ — Wed, 23 Aug 2023 19:52:14 GMT

I have a large Docker image in our AWS ECR repo. The image is 27.4 GB locally and 11539.79 MB compressed in ECR.

The error from the Event Log is:

Failed to add 2 containers to the compute. Will attempt retry: true. Reason: Docker image pull failure

JSON:

{
  "reason": {
    "code": "DOCKER_IMAGE_PULL_FAILURE",
    "type": "SERVICE_FAULT",
    "parameters": {
      "instance_id": "i-0172cf9b70a25df47",
      "databricks_error_message": "Downloading docker image has timed out"
    }
  },
  "add_node_failure_details": {
    "failure_count": 2,
    "resource_type": "container",
    "will_retry": true
  }
}

Re: Failed to start cluster: Large docker image

Michelangelo — Tue, 31 Oct 2023 14:54:26 GMT

I'm having the same issue--the official Databricks runtime GPU images are already quite large, so using them as a base causes you to run into this timeout issue. Did anyone ever find a fix?

Re: Failed to start cluster: Large docker image

amoghjain — Mon, 13 Nov 2023 21:05:40 GMT

I have a similar problem. a 10gb image pulls fine but a 31gb image doesnt. both workers and drivers have 64gb memory. i get the timeout error with "Cannot launch the cluster because pulling the docker image failed. Please double check connectivity from workers to the container registry, as well as the credentials used to pull the image"

were you able to figure out a solution?

Re: Failed to start cluster: Large docker image

Michelangelo — Thu, 16 Nov 2023 16:38:51 GMT

@Retired_mod it's not possible to change the timeout value for the Docker image pull on a Databricks cluster. That isn't exposed to the user.

Re: Failed to start cluster: Large docker image

Michelangelo — Thu, 16 Nov 2023 16:40:15 GMT

The only solution as of now is to reduce the size of your image--try a smaller base image, don't build multiple intermediate images that build off of each other, reduce the number of layers, aggressively purge apt and pip caches, etc.