cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Azure DataBricks New Job Cluster Libraries Installation Issues

karthik_elavan
New Contributor II

Dear Team,

We are trying to install runtime Libraries from Azure Data factory to Azure DataBricks and linkedservice, we are using New Job Cluster to spin the notebooks to execute the python code. we are using the third party Libraries which is prophet and needs to install while execute in runtime.

so that i am trying to add the LIB

DBFS:dbfs/FileStore/jars/prophet/ephem-4.1.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 
DBFS: dbfs/FileStore/jars/prophet/ephem-4.1.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 
DBFS:dbfs/FileStore/jars/prophet/LunarCalendar-0.0.9-py2.py3-none-any.whl

in my Azure Data Factory Append Liberaries options to spin the notebook.

below issues throwing the error while run the pipelines,

Run result unavailable: job failed with error message
 Library installation failed for library due to user error for whl: "dbfs:dbfs/FileStore/jars/prophet/setuptools_git-1.2-py2.py3-none-any.whl"
. Error messages:
Library installation attempted on the driver node of cluster 1226-011209-mj643npw and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException

3 REPLIES 3

Aviral-Bhardwaj
Esteemed Contributor III

please go with init script and share your observation after that

https://docs.databricks.com/clusters/init-scripts.html

above link will help you

Thanks

Aviral Bhardwaj

AviralBhardwaj

Meghala
Valued Contributor II

Thank you it's helpfull to me as well​

ramravi
Contributor II

init scripts are great way to handle the dependent libraries installation in cluster.

https://stackoverflow.com/questions/62516102/install-python-packages-using-init-scripts-in-a-databri...

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group