topic Re: Why can't I install python libraries when i update cluster runtime from 10.1 to 12.1? in Data Engineering

Why can't I install python libraries when i update cluster runtime from 10.1 to 12.1?

pepe — Thu, 09 Mar 2023 21:09:32 GMT

This same question was asked here 9 months ago without any answer:

https://community.databricks.com/s/question/0D58Y000096VjKrSAK/managedlibraryinstallfailed-when-changing-databricks-runtime-version-from-91-to-110

I was using runtime 9.1, and then upgraded it to 12.1, I was installing XlsxWriter using the cluster config GUI in the libraries page. It was working fine, after upgrade i can't install any libraries anymore.

With 12.1 i get this message, weird thing is that if use the web terminal and run a pip install XlsxWriter it works fine.

java.lang.RuntimeException: ManagedLibraryInstallFailed: org.apache.spark.SparkException: Cluster library installation failed on the driver because the filesystem is not ready or has been disabled. If the problem persists after a few minutes, try restarting the cluster, or contact Databricks support. for library:PythonPyPiPkgId(xlsxwriter,None,None,List()),isSharedLibrary=false

Re: Why can't I install python libraries when i update cluster runtime from 10.1 to 12.1?

Anonymous — Tue, 14 Mar 2023 23:38:56 GMT

@JOSE RODRIGUEZ : I am giving you some options to rule out, you can reply back once you have ruled the below.

Try installing other packages: See if you can install other packages using the cluster config GUI. If you're only having issues with XlsxWriter, it's possible that there's a specific issue with that package. If you're having issues with all packages, there may be a broader issue with the cluster configuration or the Databricks platform.
Check the logs: Check the logs for the cluster, driver, and worker nodes to see if there are any errors or issues related to the library installation process. This could give you more insight into what's causing the problem.
Try a different installation method: If you're not able to install packages using the cluster config GUI, try using a different installation method, such as installing the package via the web terminal or using the Databricks REST API. This may help you identify whether the issue is with the cluster configuration or the library installation process itself.

Re: Why can't I install python libraries when i update cluster runtime from 10.1 to 12.1?

Anonymous — Sat, 01 Apr 2023 00:54:18 GMT

Hi @JOSE RODRIGUEZ

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you.

Cheers!