topic Re: Cluster scoped init script failing in Data Engineering

Cluster scoped init script failing

pablobd — Fri, 23 Aug 2024 07:31:30 GMT

I am creating a cluster with asset bundles and adding a init script to it with asset bundles too. The init script is a .sh file in a UC Volume. When a I run a job, the cluster spins up and fails with this error:

Cluster '****' was terminated. Reason: INIT_SCRIPT_FAILURE (CLIENT_ERROR). Parameters: instance_id:****, databricks_error_message:Cluster scoped init script /Volumes/***.sh failed: Script exit status is non-zero.

I enabled the login to S3 and the error logged says:

bash: line 11: /Volumes/***.sh: Permission denied

The Principal spinning the cluster has ALL PRIVILEGES granted to the volume and file. The init script is quite simple (just to test that it can run):

#!/bin/bash

printf "Hello world!!"

Re: Cluster scoped init script failing

pablobd — Fri, 23 Aug 2024 07:53:39 GMT

It's actually solved, the principal that had permissions was the integ one, the job was the prod one. So, I needed to give permissions to the prod principal and now it runs.

Re: Cluster scoped init script failing

PabloCSD — Wed, 11 Sep 2024 16:15:11 GMT

Hello Pablo, where did you changed the permissons, we are having the exact same issue, but with dbx.

We are using .sh for installing a library (just doing "pip install ***.whl")