cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Azure DataBricks New Job Cluster Libraries Installation Issues

karthik_elavan
New Contributor II

Dear Team,

We are trying to install runtime Libraries from Azure Data factory to Azure DataBricks and linkedservice, we are using New Job Cluster to spin the notebooks to execute the python code. we are using the third party Libraries which is prophet and needs to install while execute in runtime.

so that i am trying to add the LIB

DBFS:dbfs/FileStore/jars/prophet/ephem-4.1.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 
DBFS: dbfs/FileStore/jars/prophet/ephem-4.1.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 
DBFS:dbfs/FileStore/jars/prophet/LunarCalendar-0.0.9-py2.py3-none-any.whl

in my Azure Data Factory Append Liberaries options to spin the notebook.

below issues throwing the error while run the pipelines,

Run result unavailable: job failed with error message
 Library installation failed for library due to user error for whl: "dbfs:dbfs/FileStore/jars/prophet/setuptools_git-1.2-py2.py3-none-any.whl"
. Error messages:
Library installation attempted on the driver node of cluster 1226-011209-mj643npw and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException

3 REPLIES 3

Aviral-Bhardwaj
Esteemed Contributor III

please go with init script and share your observation after that

https://docs.databricks.com/clusters/init-scripts.html

above link will help you

Thanks

Aviral Bhardwaj

Meghala
Valued Contributor II

Thank you it's helpfull to me as wellโ€‹

ramravi
Contributor II

init scripts are great way to handle the dependent libraries installation in cluster.

https://stackoverflow.com/questions/62516102/install-python-packages-using-init-scripts-in-a-databri...

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.