Unable to Install Cluster-Scoped Libraries on Runtime >13.3

Esther_Tomi — Mon, 15 Jan 2024 13:23:56 GMT

Hello team,

I'm trying to upgrade our databricks runtime to 13.3 from 9.1, but i've been having issues installing libraries on the compute from our internal artifactory

However, when I tried this on a unity-catalog enabled workspace, it works seamlessly, but the thing is, we don't have unity catalog on our workspace and probably not ready for that move now.

Is there a possibility to install compute-scoped library from internal artifactory without a unity catalog on our workspace?

The error i keep getting is:

Library installation attempted on the driver node of cluster xxx and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: org.apache.spark.SparkException: Process List(/bin/su, libraries, -c, bash /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip install 'simplejson' --disable-pip-version-check) exited with code 1. WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /simple/simplejson/WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /simple/simplejson/WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /simple/simplejson/WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /simple/simplejson/WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /simple/simplejson/ERROR: Could not find a version that satisfies the requirement simplejson (from versions: none)ERROR: No matching distribution found for simplejson

topic Unable to Install Cluster-Scoped Libraries on Runtime >13.3 in Data Engineering

Unable to Install Cluster-Scoped Libraries on Runtime >13.3