PyPI cluster libraries failing to get installed

unauthenticated — Thu, 29 Feb 2024 11:09:35 GMT

Hi all,

In my cluster, some of the PyPI cluster libraries started failing to get installed. It is weird because some of them get installed and some of are constantly failing. In every failed one, the error message is the same (just a package name is different):

"Library installation attempted on the driver node of cluster XXX and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: org.apache.spark.SparkException: Process List(/bin/su, libraries, -c, bash /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip install 'jaydebeapi' --disable-pip-version-check) exited with code 1. WARNING: The directory '/home/libraries/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag."

Recently, I turned on Table Access Control for Hive metastore, which required switching cluster access mode from "No isolation shared" to "Shared". However, even after switching back to "No isolation shared", the above problem still persists.

I can't see what could possibly cause this problem, and I can't find any solution. Any tips/advices/etc. are helpful.

Thanks.

Re: PyPI cluster libraries failing to get installed

jacovangelder — Wed, 13 Mar 2024 14:11:26 GMT

I've had this issue myself. What ended up to be the problem is I had windows line endings in my .sh script. You need to convert them to Linux line endings.

topic PyPI cluster libraries failing to get installed in Administration & Architecture

PyPI cluster libraries failing to get installed

Re: PyPI cluster libraries failing to get installed