Init Script Failing

[semsim]

I am getting an error when I try to run the cluster scoped init script. The script itself is as follows:

#!/bin/bash

sudo apt update && sudo apt upgrade -y

sudo apt install libreoffice-common libreoffice-java-common libreoffice-writer openjdk-8-jre-headless -y

below you can find the error message:

Init script failure:
Cluster scoped init script <path to .sh file> failed: Script exit status is non-zero

I've tried uploading the .sh via databricks CLI. I have also read about enabling logging which would be nice to get some help with.

Thanks for your help and time. 

[SashankKotta]

It is failing with exit status non-zero, which means the issue is in the init script file; please ensure there are no control-m chars in your init script sh file. you can run 

dos2unix <init_scrip_shell_file>

 or manually remove control-m chars in vi editor and upload. It should work.

Ideally, It is recommended to use custom compute cluster policies for any library installation instead of init scripts.

[jacovangelder]

Most likely your init script has windows line endings.
Open it in VSCode and change the line endings to Unix (LF) line endings. 

[semsim]

@jacovangelder @SashankKotta ,

Is that all it could be? I've double checked the line endings and it is LF with no ^M chars. I took the file directly from WSL and uploaded it to DBFS.and it is still failing with a non-zero exit status.

[jacovangelder]

Did some debugging for you. It works for me using this

#!/bin/bash

sudo apt update && sudo apt upgrade -y
sudo apt-get install libreoffice-common libreoffice-java-common libreoffice-writer openjdk-8-jre-headless -y

Not sure why apt get works and apt install doesn't
Btw, Init scripts on DBFS are being deprecated any moment now. Best update to Workspace files or a UC volume. 

[semsim]

@jacovangelder  That does not seem to do it for me either. I have updated the location of the init script to be a workspace location rather than DBFS.

[jacovangelder]

You're right, it doesn't work. Not sure why my cluster was showing up as running last night, most likely something caching related. I'll try to jump into it at some later point if I have some time. Very strange!

[semsim]

Could you elaborate on the caching idea?

Thanks

[semsim · Accepted solution]

The following code solved my problem: 

sudo rm -r /var/lib/apt/lists/* 
sudo apt clean && sudo apt update --fix-missing -y
*add apt statement for library you're trying to install*

This allowed me to successfully spin up the cluster every time using init scripts.

[zmsoft]

Hi @semsim , @jacovangelder ,

I added the code you mentioned at the beginning of the script, but I still got errors.

#!/bin/bash
sudo rm -r /var/lib/apt/lists/* 
sudo apt clean && sudo apt update --fix-missing -y
if ! [[ "18.04 20.04 22.04 23.04 24.04" == *"$(lsb_release -rs)"* ]];
then
    echo "Ubuntu $(lsb_release -rs) is not currently supported.";
    exit;
fi

# Add the signature to trust the Microsoft repo
# For Ubuntu versions < 24.04 
curl https://packages.microsoft.com/keys/microsoft.asc | sudo tee /etc/apt/trusted.gpg.d/microsoft.asc
# For Ubuntu versions >= 24.04
curl https://packages.microsoft.com/keys/microsoft.asc | sudo gpg --dearmor -o /usr/share/keyrings/microsoft-prod.gpg

# Add repo to apt sources
curl https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/prod.list | sudo tee /etc/apt/sources.list.d/mssql-release.list

# Install the driver
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18
# optional: for bcp and sqlcmd
sudo ACCEPT_EULA=Y apt-get install -y mssql-tools18
echo 'export PATH="$PATH:/opt/mssql-tools18/bin"' >> ~/.bashrc
source ~/.bashrc
# optional: for unixODBC development headers
sudo apt-get install -y unixodbc-dev

sudo apt-get install python3-pip -y
pip3 install --upgrade pyodbc

Error msg : Script exit status is non-zero.

Any suggestions?