init_script breaks Notebooks

[stadelmannkevin]

Hi everyone

We would like to use our private company Python repository for installing Python libraries with pip install.

To achieve this, I created a simple script which sets the index-url configuration of pip to our private repo

I set this script as an init_script for my personal compute cluster.

When I start the cluster, the script loads successfully, and I can see that the two lines are correctly added to the pip.conf file.

However, here's the issue: I am unable to execute any Python/Spark/SQL/R commands from a notebook attached to this cluster. As a result, pip install <library> also doesn't work in the notebook.

Interestingly, when I run pip install directly in the cluster's web terminal, it works perfectly as intended.

Has anyone encountered similar issues?

PS: Just tested: any init_script will cause the same issue, even if the init script is empty! I'm not able to execute any language commands in a attacked notebook.

[Walter_C]

Does it throws any error message or it just hangs? Have you tried additional DBRs on the cluster to see if this is an issue with the DBR version being run or general issue?

[PabloCSD]

I had a similar issue, I installed a private library and I could not use the cluster in a notebook for any commands.

I fixed it cloning the cluster, it was like the cluster was saturated. The error I got was something like this:

Failure starting repl. Try detaching and re-attaching the notebook.

Other alternative is to install the new libraries in the notebook instead of using an init_script in the first cells.

Hope it works!