topic Re: Azure DataBricks New Job Cluster Libraries Installation Issues in Data Engineering

Azure DataBricks New Job Cluster Libraries Installation Issues

karthik_elavan — Mon, 26 Dec 2022 01:33:25 GMT

Dear Team,

We are trying to install runtime Libraries from Azure Data factory to Azure DataBricks and linkedservice, we are using New Job Cluster to spin the notebooks to execute the python code. we are using the third party Libraries which is prophet and needs to install while execute in runtime.

so that i am trying to add the LIB

DBFS:dbfs/FileStore/jars/prophet/ephem-4.1.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 
DBFS: dbfs/FileStore/jars/prophet/ephem-4.1.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 
DBFS:dbfs/FileStore/jars/prophet/LunarCalendar-0.0.9-py2.py3-none-any.whl

in my Azure Data Factory Append Liberaries options to spin the notebook.

below issues throwing the error while run the pipelines,

Run result unavailable: job failed with error message
 Library installation failed for library due to user error for whl: "dbfs:dbfs/FileStore/jars/prophet/setuptools_git-1.2-py2.py3-none-any.whl"
. Error messages:
Library installation attempted on the driver node of cluster 1226-011209-mj643npw and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException

Re: Azure DataBricks New Job Cluster Libraries Installation Issues

Aviral-Bhardwaj — Mon, 26 Dec 2022 04:38:33 GMT

please go with init script and share your observation after that

https://docs.databricks.com/clusters/init-scripts.html

above link will help you

Thanks

Aviral Bhardwaj

Re: Azure DataBricks New Job Cluster Libraries Installation Issues

Meghala — Mon, 26 Dec 2022 14:39:29 GMT

Thank you it's helpfull to me as well

Re: Azure DataBricks New Job Cluster Libraries Installation Issues

ramravi — Mon, 02 Jan 2023 14:32:49 GMT

init scripts are great way to handle the dependent libraries installation in cluster.

https://stackoverflow.com/questions/62516102/install-python-packages-using-init-scripts-in-a-databricks-cluster