Hello everyone,

I get a weird error when installing additional libraries in my cluster.

I have a predefined Databricks cluster (Standard_L8s_v2) as a Compute instance. I run pipelines on that cluster in Azure ADF. The pipeline consists several tasks. The tasks run Python code.

I install my Python code with a prebuilt wheel. Additionally I need to add four more libraries to the Tasks - Settings - Additional Libraries and install them with pip. This step is necessary, so that pytorch (one of the four libraries) is installed with GPU support, as the libraries and dependencies of the wheel are defined with poetry. 

But the library installation fails regularly. It does not always fail for the same task on the same day. Sometimes it fails for Task1 on day1 and the other day for Task2 on day2. Sometimes all succeed and sometimes all fail.

Here's the error message:

run failed with error message Library installation failed for library due to user error for pypi { package: "sentence-transformers==2.2.2" } Error messages: Library installation failed after PENDING for 10 minutes since cluster entered RUNNING state. Error Code: CHAUFFEUR_RPC_SERVER_UNAVAILABLE. Library request cannot reach driver node on cluster 0511-114900-l5r08j93. This could be caused by network connectivity to the driver node being temporarily down. If this doesn't self correct in a while, please check your network settings or contact Databricks Support.

What I suspected that the configuration of the cluster: Terminate after 10 minutes of inactivity. The assumption I had was that the cluster is not in RUNNING state during the time of installing the libraries in the appended libraries section. Does that make sense?

I increased the time to 20 and 30 minutes but it still sometimes fails. It seems works more stable when increasing it to 40 minutes. But the results I have here are not really validated. It also more regularly fails if the cluster is triggered by an automatic trigger than when starting the pipeline manually (I don't understand why).

Does anyone have an idea why the library installation fails? Let me know if you need further context!

Thanks for your help. Really appreciated!