Hi all,
In my cluster, some of the PyPI cluster libraries started failing to get installed. It is weird because some of them get installed and some of are constantly failing. In every failed one, the error message is the same (just a package name is different):
"Library installation attempted on the driver node of cluster XXX and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: org.apache.spark.SparkException: Process List(/bin/su, libraries, -c, bash /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip install 'jaydebeapi' --disable-pip-version-check) exited with code 1. WARNING: The directory '/home/libraries/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag."
Recently, I turned on Table Access Control for Hive metastore, which required switching cluster access mode from "No isolation shared" to "Shared". However, even after switching back to "No isolation shared", the above problem still persists.
I can't see what could possibly cause this problem, and I can't find any solution. Any tips/advices/etc. are helpful.
Thanks.
