topic Re: Installing libraries on job clusters in Administration & Architecture

Installing libraries on job clusters

erigaud — Tue, 11 Jul 2023 07:09:46 GMT

Simple question : what is the way to go to install libraries on job clusters ? There does not seem to be a "Libraries" tab on the UI as opposed to regular clusters. Does it mean that the only option is to use init scripts ?

Re: Installing libraries on job clusters

ranged_coop — Tue, 11 Jul 2023 09:54:22 GMT

In the job, you would have the dependent library option, where you can mention the libraries you need installed. They will get installed on the job cluster before the job starts. This link might be useful...

Re: Installing libraries on job clusters

Tharun-Kumar — Wed, 12 Jul 2023 04:45:56 GMT

@erigaud

You can traverse to the Tasks section of the job and to the first task in your job. Under the task properties, you would be seeing Dependent libraries using which you can install libraries from maven or python or even a custom JAR.

Re: Installing libraries on job clusters

erigaud — Wed, 12 Jul 2023 06:00:24 GMT

Will installing a lib on a task for a job make this lib available for all the other tasks aswell ? (assuming they all use the same cluster)

Re: Installing libraries on job clusters

Tharun-Kumar — Wed, 12 Jul 2023 07:21:38 GMT

@erigaud Yes it will be available for all the subsequent tasks on the cluster.

Re: Installing libraries on job clusters

MouhannadOweis — Wed, 24 Apr 2024 21:44:27 GMT

How can you do this in ADF?

Re: Installing libraries on job clusters

daniel_sahal — Thu, 25 Apr 2024 05:14:02 GMT

@MouhannadOweis
You can also add libraries to ADF Databricks task.

Re: Installing libraries on job clusters

brushdoylt — Fri, 26 Apr 2024 07:31:07 GMT

@daniel_sahalflappy bird

Is there another way to add libraries to the ADF Databricks task even faster?

Re: Installing libraries on job clusters

MouhannadOweis — Fri, 26 Apr 2024 15:54:18 GMT

Thank you that worked.

Re: Installing libraries on job clusters

imsabarinath — Mon, 15 Jul 2024 07:31:25 GMT

You may want to copy required libs to a volume and load it during cluster setup to avoid downloading the libs for every run.

Re: Installing libraries on job clusters

cleversuresh — Fri, 03 Jan 2025 03:23:35 GMT

I am not able to select the requirements.txt file from my workspace folder, I can see the file but cannot select it. How do I overcome this problem?

Re: Installing libraries on job clusters

sachamourier — Tue, 25 Feb 2025 21:16:22 GMT

Do you know how to do that ? I have tried to upload some requirements.txt and some .whl files to my unity catalog volume, using the init script below but the job cluster library installation is very unconsistent...
---

#!/bin/bash

if [ "$ENV" = "dev" ]; then

pip install "/Volumes/.../core_utils-0.2-py3-none-any.whl"

pip install -r "/Volumes/.../requirements.txt"

elif [ "$ENV" = "qual" ]; then

pip install "/Volumes/.../core_utils-0.2-py3-none-any.whl"

pip install "/Volumes/.../requirements.txt"

pip install -r "/Volumes/.../requirements.txt"

Re: Installing libraries on job clusters

Charlenendoza — Wed, 27 Aug 2025 09:36:56 GMT

Frankly, I'm also scratching my head about this library installation conundrum in job clusters; the missing Libraries tab is definitely throwing me off. Seems like init scripts are the only officially sanctioned route, which feels a bit clunky. Are there any more elegant solutions people have discovered, perhaps involving containerization or custom images? Anyone have experience with managing dependencies this way on job clusters?

Re: Installing libraries on job clusters

smourier — Wed, 27 Aug 2025 09:53:10 GMT

Hello,

One solution I have found is to use job cluster Policies. With that, you are able to design groups of job clusters, where libraries can be installed just as in all-purpose computes.

In my case, I needed these job clusters to run through Azure Data Factory and all I had to do once I defined my policies, was to add the policy ID to my Databricks linked service.

Hope this helps,

Sacha