Databricks Asset Bundles library dependencies - JAR file

theanhdo — Thu, 11 Jul 2024 00:39:53 GMT

Hi there,

I have used databricks asset bundles (DAB) to deploy workflows. For each job, I will create a job cluster and install external libraries by specifying libraries in each task, for example:

- task_key: my-task
job_cluster_key: my-cluster
notebook_task:
notebook_path: ../notebooks/my_notebook.ipynb
libraries:
- whl: /Workspace${workspace.file_path}/libraries/PyYAML-6.0.whl
- jar: /Workspace${workspace.file_path}/libraries/mongo-spark-connector_2.12-10.1.1-all.jar

For Python wheel file (whl), I can see the library PyYAML-6.0.whl is installed. However, for JAR file, it failed to install the library mongo-spark-connector_2.12-10.1.1-all.jar. For JAR file, I know that I can install it using Unity Catalog volumes, however I want to install all libraries from workspace.

From this document, it says that To add a JAR file to a job task, in libraries specify a jar mapping for each library to be installed. You can install a JAR from workspace files, Unity Catalog volumes, cloud object storage, or a local file path.
https://docs.databricks.com/en/dev-tools/bundles/library-dependencies.html
However, it is not working for JAR file when I use workspace to install JAR file.

Even when I created a cluster and tried to install JAR file by the following steps, it is not working:
1. Create a cluster using Databricks Runtime Version 14.3 LTS
2. Go to Libraries tab and click Install new button
3. In the popup, select Workspace and navigate to the workspace libraries folder
Then there are 2 libraries PyYAML-6.0.whl and mongo-spark-connector_2.12-10.1.1-all.jar. I can only select whl library while JAR library is not selectable.

Do you know is there any way we can install JAR files from workspace?

Re: Databricks Asset Bundles library dependencies - JAR file

theanhdo — Mon, 15 Jul 2024 23:57:22 GMT

Thanks very much @Retired_mod for your thorough answer.

topic Re: Databricks Asset Bundles library dependencies - JAR file in Data Engineering

Databricks Asset Bundles library dependencies - JAR file

Re: Databricks Asset Bundles library dependencies - JAR file