Azure DataBricks New Job Cluster Libraries Installation Issues
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-25-2022 05:33 PM
Dear Team,
We are trying to install runtime Libraries from Azure Data factory to Azure DataBricks and linkedservice, we are using New Job Cluster to spin the notebooks to execute the python code. we are using the third party Libraries which is prophet and needs to install while execute in runtime.
so that i am trying to add the LIB
DBFS:dbfs/FileStore/jars/prophet/ephem-4.1.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
DBFS: dbfs/FileStore/jars/prophet/ephem-4.1.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
DBFS:dbfs/FileStore/jars/prophet/LunarCalendar-0.0.9-py2.py3-none-any.whlin my Azure Data Factory Append Liberaries options to spin the notebook.
below issues throwing the error while run the pipelines,
Run result unavailable: job failed with error message
Library installation failed for library due to user error for whl: "dbfs:dbfs/FileStore/jars/prophet/setuptools_git-1.2-py2.py3-none-any.whl"
. Error messages:
Library installation attempted on the driver node of cluster 1226-011209-mj643npw and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException
- Labels:
-
Azure
-
Azure databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-25-2022 08:38 PM
please go with init script and share your observation after that
https://docs.databricks.com/clusters/init-scripts.html
above link will help you
Thanks
Aviral Bhardwaj
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-26-2022 06:39 AM
Thank you it's helpfull to me as well
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-02-2023 06:32 AM
init scripts are great way to handle the dependent libraries installation in cluster.