Dask-Databricks init script not working

[rbauer]

Hello everybody ! 

I am trying to use the Dask-Databricks distribution (https://github.com/dask-contrib/dask-databricks?tab=readme-ov-file)

i set up the required init-script according to the instructions on the Github page and had no problems there, however the cluster is not able to start always crashing with this error message : failed: Script exit status is non-zero

Unfortunately i don't know the first thing about scripting or init scripts and couldn't find anything to fix this

This is the script i used : 

#!/bin/bash

# Install Dask + Dask Databricks
/databricks/python/bin/pip install --upgrade dask[complete] dask-databricks

# Start Dask cluster components
dask databricks run

Thank you for your help ! 

[Kaniz]

Hi @rbauer, It seems you’re encountering issues with starting your Dask-Databricks cluster due to a non-zero exit status from your init script.

Let’s troubleshoot this together.

Here are some steps you can take to address the problem:

1. Check the Init Script:

   • Ensure that the init script you’ve set up follows the correct syntax and is free of errors.
   • Verify that the path to the script is accurate. You’ll need the full path to the location of the script (e.g., dbfs:/databricks/<init-script-folder>/create_dummy_file.sh).

2. Cluster-Scoped Init Script:

   • Consider using a cluster-scoped init script instead of a global or cluster-named one. Cluster-scoped init scripts avoid synchronous blocking of RPCs during execution, which can help prev...¹.

3. Mirror Sync in Progress:

   • If your init script references a problematic repository, wait for the mirror to finish synchronizing with the repository before starting your cluster. Alternatively, remove the reference to the problematic repo from your init script².

4. Remove Extra Quotes:

   • Sometimes, extra quotes can cause init script failures. Make sure there are no unnecessary double quotes in your script. Removing them might resolve the issue³.

If you encounter further difficulties, feel free to ask for more assistance. Good luck, and I hope your Dask cluster starts smoothly! 🚀🌟