Hello,

We have an Azure Data Factory pipeline running during the night, and one of the activities calls a Databricks Notebook with dynamic DatabricksInstancePoolId, ClusterVersion and Workers. Yesterday, it failed with with the following error:

Cluster terminated. Reason: Init script failure

Cluster scoped init script dbfs:/InitScripts/InstallODBCDriverForSQLServer.sh failed: Script exit status is non-zero

I checked the run in databricks and all I had in the output was:

Run result unavailable: job failed with error message

Unexpected failure while waiting for the cluster Some((xxxxxxxxxxxxxxx) )to be readySome(: Cluster xxxxxxxxxxxxxxxxis in unexpected state Terminated: INIT_SCRIPT_FAILURE(CLIENT_ERROR)instance_id:xxxxxxxxxxxxx,databricks_error_message:Cluster scoped init script dbfs:/InitScripts/InstallODBCDriverForSQLServer.sh failed: Script exit status is non-zero.).

Back in Azure Data Factory, I rerun the pipeline from failed activity and it succeeded, but I'm not sure how to prevent this from happening again.

Any idea how to troubleshoot this issue?