topic Re: How to grant custom container AWS credentials for reading init script? in Get Started Discussions

How to grant custom container AWS credentials for reading init script?

mrstevegross — Thu, 23 Jan 2025 19:25:54 GMT

I'm using a customer container *and* init scripts. At runtime, I get this error:

Cluster '...' was terminated. Reason: INIT_SCRIPT_FAILURE (CLIENT_ERROR). Parameters: instance_id:i-0440ddd3a2d5cce79, databricks_error_message:Cluster scoped init script s3://<our_bucket>/<our_init_script.sh> failed: Timed out with exception after 5 attempts (debugStr = 'Reading remote file for init script'), Caused by: com.databricks.objectstore.location.PermanentStorageException$AwsForbidden: Missing credentials to access AWS bucket.

It worked previously *without* the container, so I'm pretty sure the use of container is triggering the problem. I suspect that the fetch-init-scripts-from-s3 operation is occurring *inside* the container, and that the container itself lacks AWS credentials. What's the preferred way to pass in AWS credentials to a custom container?

Re: How to grant custom container AWS credentials for reading init script?

Isi — Thu, 23 Jan 2025 22:51:30 GMT

Hey! It would be great if you could share more details about the cluster type and access mode.

If you are using, for example, an all-purpose cluster with shared access mode, I recommend configuring the "Init Script" option inside the advanced cluster settings.

If Unity Catalog is enabled, ensure that your S3 path (`s3://<our_bucket>/<our_init_script.sh>`) is allowed under Catalog Explorer > Allowed Jars/init Scripts.

It would also be helpful to understand why you want to use the init script, as there might be other options available.

I hope you find this helpful. 🙂

Re: How to grant custom container AWS credentials for reading init script?

mrstevegross — Mon, 27 Jan 2025 16:01:13 GMT

>If you are using, for example, an all-purpose cluster with shared access mode, I recommend configuring the "Init Script" option inside the advanced cluster settings.

Yep, that's the approach. I've got init scripts specified in the cluster settings, and am encountering the "Missing credentials to access AWS bucket" when my job runs.

>It would also be helpful to understand why you want to use the init script, as there might be other options available.

We need to a variety of variables that are only known when the job starts, as well as start up some processes on the container (mostly for telemetry/logging retention).

Re: How to grant custom container AWS credentials for reading init script?

mrstevegross — Mon, 27 Jan 2025 18:29:04 GMT

Followup: I got the AWS creds working by amending our AWS role to permit read/write access to our S3 bucket. Woohoo!