Library installation failed for library due to user error for pypi

AlexMc
New Contributor III

‎07-14-2025 04:42 AM

Hi!

I get the below error when a cluster job starts up and tries to install a Python .whl file. (Which is hosted on an Azure Artefact feed, though this seems more like a problem of trying to read from a disk/network storage). The failure is seemingly random and intermittent, from the error message it is clearly a networking/timeout problem.

I see in the log below it mentions Retry(total=4 ... Is it possible to increase/modify this? Or perhaps adds some exponential backoff?

Thanks!
Alex

Library installation attempted on the driver node of cluster xxxxxxxx and failed. Please refer to the following error message or contact Databricks support. Error code: FAULT_OTHER, error message: org.apache.spark.SparkException: Process List(/bin/su, libraries, -c, bash /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip install 'my.company.library==1.0.0' --disable-pip-version-check) exited with code 1.   WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f872e82bf40>, 'Connectio ...

Labels:
0 Kudos

szymon_dybczak
Esteemed Contributor III

‎07-14-2025 04:56 AM - edited ‎07-14-2025 04:59 AM

Hi @AlexMc ,

You can try to increase timeout and retries number using pip command line options: 

pip install 'your_library' \
  --timeout 300 \
  --retries 10 \
  --disable-pip-version-check

 

0 Kudos

Pat
Esteemed Contributor

‎07-14-2025 05:06 AM

I would check the network connection between cluster and repository. 
The error shows pip is currently in the retry phase due to a ConnectTimeoutError, indicating network connectivity issues when trying to reach the package repository.

0 Kudos

AlexMc
New Contributor III

‎07-14-2025 06:29 AM

Thanks both - I think the problem is that this library installation is called when creating a new Job & Task via the rest endpoint. Where the libraires are specified in the .json file. 

So short version, don't think I can 'get at' the pip install call in order to add extra parameters. Instead sounds like I might have to remove the libraries from the .json and install at the notebook level with a %pip command. (Where I have more control of the retry logic)