PyPI cluster libraries failing to get installed

[unauthenticated]

Hi all,

In my cluster, some of the PyPI cluster libraries started failing to get installed. It is weird because some of them get installed and some of are constantly failing. In every failed one, the error message is the same (just a package name is different): 

"Library installation attempted on the driver node of cluster XXX and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: org.apache.spark.SparkException: Process List(/bin/su, libraries, -c, bash /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip install 'jaydebeapi' --disable-pip-version-check) exited with code 1. WARNING: The directory '/home/libraries/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag."

Recently, I turned on Table Access Control for Hive metastore, which required switching cluster access mode from "No isolation shared" to "Shared". However, even after switching back to "No isolation shared", the above problem still persists.

 I can't see what could possibly cause this problem, and I can't find any solution. Any tips/advices/etc. are helpful.

Thanks.

[Kaniz]

Hi @unauthenticated , It appears that you’re encountering issues with PyPI cluster libraries installation in your Databricks cluster.

Let’s explore some potential solutions to address this problem:

1. Check Cluster Logs:

   • Start by examining the cluster logs to identify any error messages or exceptions related to library installation. These logs can provide valuable insights into what might be going wrong.
   • Look for any specific details about the failed installation attempts, such as missing dependencies or permission issues.

2. Manual Installation:

   • Try manually installing the problematic library on the cluster to see if it succeeds.
   • You can do this by running the installation command directly on the cluster. For example, execute the following command in a notebook cell:

     !pip install jaydebeapi
     

   • Observe whether the library installs successfully or encounters any errors.

3. Permissions and Ownership:

   • The error message you provided mentions a directory permission issue: “The directory ‘/home/libraries/.cache/pip’ or its parent directory is not owned or is not writable by the current user.”
   • Ensure that the user executing the installation has the necessary permissions to write to the specified directories.
   • Verify ownership and permissions for the relevant directories (/home/libraries/.cache/pip and its parent).

4. Cache Considerations:

   • The warning about cache being disabled suggests that there might be issues related to caching.
   • Check if there are any issues with the cache setup or if it needs to be cleared.
   • Investigate whether the cache directory is accessible and writable.

5. Cluster Access Mode:

   • You mentioned switching the cluster access mode from “No isolation shared” to “Shared” due to enabling Table Access Control for Hive metastore.
   • Even after switching back to “No isolation shared,” the problem persists.
   • Double-check that the cluster configuration is correctly set to the desired access mode.
   • Restart the cluster after making any changes to ensure they take effect.

I hope these tips help you identify and resolve the issue! 🚀

[jacovangelder]

I've had this issue myself. What ended up to be the problem is I had windows line endings in my .sh script. You need to convert them to Linux line endings.