topic Re: Format when specifying docker_image url? in Get Started Discussions

Format when specifying docker_image url?

mrstevegross — Tue, 21 Jan 2025 21:06:29 GMT

I am providing a custom Docker image to my Databricks/Spark job. I've created the image and uploaded it to our private ECR registry (the URL is `472542229217.dkr.ecr.us-west-2.amazonaws.com/tectonai/mrstevegross-testing:latest`). Based on the docs (https://docs.databricks.com/en/compute/custom-containers.html#launch-your-compute-using-the-api), however, it is unclear to me if my string format is correct, since I'm getting this error at runtime:

Cluster '0121-203731-epwa9mar' was terminated. Reason: INVALID_ARGUMENT (CLIENT_ERROR). Parameters: databricks_error_message:Container setup failed because of an invalid request: Exception when verifying docker container image: Image doesn't exist or invalid credential to pull image from 472542229217.dkr.ecr.us-west-2.amazonaws.com/tectonai/mrstevegross-testing:latest. Stdout: Stderr: time="2025-01-21T20:43:05Z" level=fatal msg="Error parsing image name \"docker://472542229217.dkr.ecr.us-west-2.amazonaws.com/tectonai/mrstevegross-testing:latest\": reading manifest latest in 472542229217.dkr.ecr.us-west-2.amazonaws.com/tectonai/mrstevegross-testing: authentication required"

Note in particular that the error claims "parsing image name 'docker://..."; I'm wondering whether the "docker://" prefix indicates that I mis-specified the URL. Can anyone advise on how to properly format "472542229217.dkr.ecr.us-west-2.amazonaws.com/tectonai/mrstevegross-testing:latest" for use in the API call?

Re: Format when specifying docker_image url?

mrstevegross — Tue, 21 Jan 2025 21:13:07 GMT

Note: I found additional docs about formatting for AWS ECR here, so, I'm trying that now.

Re: Format when specifying docker_image url?

Isi — Thu, 23 Jan 2025 23:49:16 GMT

Hey!

It seems like your Instance Profile might not have enough privileges to access this ECR. I would recommend updating the policies of the IAM role you are using and ensuring that it includes at least the following permissions:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "GrantECRGeneralAccess", "Effect": "Allow", "Action": [ "ecr:GetRegistryPolicy", "ecr:DescribeRegistry", "ecr:GetAuthorizationToken" ], "Resource": "<resource>" }, { "Sid": "GrantECRReadWriteAccess", "Effect": "Allow", "Action": [ "ecr:DescribeImageScanFindings", "ecr:GetLifecyclePolicyPreview", "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "ecr:DescribeImageReplicationStatus", "ecr:DescribeImages", "ecr:DescribeRepositories", "ecr:ListTagsForResource", "ecr:ListImages", "ecr:BatchCheckLayerAvailability", "ecr:GetRepositoryPolicy", "ecr:GetLifecyclePolicy", "ecr:InitiateLayerUpload", "ecr:SetRepositoryPolicy", "ecr:PutImageTagMutability", "ecr:StartImageScan", "ecr:UploadLayerPart", "ecr:BatchDeleteImage", "ecr:CompleteLayerUpload", "ecr:TagResource", "ecr:ReplicateImage", "ecr:PutLifecyclePolicy", "ecr:PutImageScanningConfiguration", "ecr:PutImage", "ecr:UntagResource", "ecr:StartLifecyclePolicyPreview" ], "Resource": "<resource>" } ] }

After updating the IAM role with the above permissions, make sure the Instance Profile ARN is correctly assigned to your Databricks cluster.

🙂

Re: Format when specifying docker_image url?

mrstevegross — Mon, 27 Jan 2025 16:02:52 GMT

Thanks, that's pretty much what I did; a lot of terraform configuration to get the AWS account set up properly, and now I'm able to tell DBR to load the container. (FWIW, I'm encountering *new* access issues; I started a thread here (https://community.databricks.com/t5/community-platform-discussions/how-to-grant-custom-container-aws-credentials-for-reading-init/m-p/106856) to deal with them).