Unable to Install Cluster-Scoped Libraries on Runtime >13.3

[Esther_Tomi]

Hello team,

I'm trying to upgrade our databricks runtime to 13.3 from 9.1, but i've been having issues installing libraries on the compute  from our internal artifactory

However, when I tried this on a unity-catalog enabled workspace, it works seamlessly, but the thing is, we don't have unity catalog on our workspace and probably not ready for that move now.

Is there a possibility to install compute-scoped library from internal artifactory without a unity catalog on our workspace? 

The error i keep getting is:

 Library installation attempted on the driver node of cluster xxx and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: org.apache.spark.SparkException: Process List(/bin/su, libraries, -c, bash /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip install 'simplejson' --disable-pip-version-check) exited with code 1. WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /simple/simplejson/WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /simple/simplejson/WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /simple/simplejson/WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /simple/simplejson/WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /simple/simplejson/ERROR: Could not find a version that satisfies the requirement simplejson (from versions: none)ERROR: No matching distribution found for simplejson 

[Kaniz_Fatma]

Hi @Esther_Tomi, According to a StackOverflow post, you can still use third-party libraries as notebook-scoped libraries in Databricks without Unity Catalog.

You can install libraries in a notebook using the - 

 %python pip install library_of_interest command.

[Kaniz_Fatma]

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!