Cluster cannot find init script stored in Volume

[jimoskar]

I have created an init script stored in a Volume which I want to execute on a cluster with runtime 16.4 LTS. The cluster has policy = Unrestricted and Access mode = Standard. I have additionally added the init script to the allowlist. This should be sufficient per the documentation. However, when I try to start the cluster, I get

cannot execute: required file not found

when I try to start the cluster. Anyone who knows how to fix this? I am the owner of the volume where the script is stored, so access should not be an issue.

[szymon_dybczak]

Hi @jimoskar ,

Since you're using standard access mode you need to add init script to allowlist. Did you add your init script to allowlist? If not, do the following:

1. In your Databricks workspace, click  Catalog.

2. Click the gear icon .
3. Click the metastore name to open the metastore details and permissions UI.
4. Select Allowed JARs/Init Scripts.
5. Click Add.

Next add an init script to the allowlist perform following steps:

1. For Type, select Init Script.
2. For Source Type, select Volume or the object storage protocol.
3. Specify the source path to add to the allowlist. See How are permissions on paths enforced in the allowlist?.

[szymon_dybczak]

Hi @jimoskar ,

Another possibility I can think of is that during cluster startup, the init script path is resolved using the cluster owner's privileges. If the owner lacks access to the volume or script, the initialization fails even if the user starting the cluster does have the necessary permissions.

To identify the creator or owner of a cluster: 

1. Navigate to the cluster configuration UI. 
2. Click the kebab menu and select JSON. 
3. In the JSON view, look for the field creator_user_name to find the creator’s username. 
    

To grant the necessary permissions on the volume: 

1. Navigate to the Catalog Explorer.
2. Click on the catalog that contains the volume.
3. Go to the Permissions tab.
4. Click the Grant button.
5. Assign the required permissions on the volume to the creator.

The following image shows the UI after navigating to the Permissions tab and highlights the Grant button location on the screen.

[jimoskar]

Hi @szymon_dybczak,

Thank you for your suggestions! I have added the init script to the allowlist. Additionally, my user is the creator of the compute, and has ALL PRIVELEGES on the Catalog where the init script is stored. However, I still get the same error.

Do you have any other suggestions for possible solutions?

[szymon_dybczak]

Hi @jimoskar ,

Yep, I have some other ideas that we can check.  In databricks you have 2 kinds of init scripts:

- global init scripts - a global init script runs on all clusters in your workspace configured with dedicated (formerly single user) or legacy no-isolation shared access mode. So it seems that global init script are supported only for those access mode.

- Cluster-scoped init scripts - Cluster-scoped init scripts are init scripts defined in a cluster configuration. Cluster-scoped init scripts apply to both clusters you create and those created to run jobs.

So, do you happen to use global init script? Because it could be one explanation why it doesn't work in your case.

Another thing to check - what is the size of your init script? In docs they mentioned that:

"The init script cannot be larger than 64KB. If a script exceeds that size, the cluster will fail to launch and a failure message will appear in the cluster log."

Also, you can try to check init script logs. Maybe we will find some additional info/error message there?

And last, but not least you can check below articles. It seems that Windows Notepad and similar editors insert special characters like carriage returns (\r), that could cause some issues:

Init scripts failing with unexpected end of file error - Databricks

Init script stored on a volume fails to execute on cluster start - Databricks