Installing libraries on job clusters

[erigaud]

Simple question : what is the way to go to install libraries on job clusters ? There does not seem to be a "Libraries" tab on the UI as opposed to regular clusters. Does it mean that the only option is to use init scripts ? 

[ranged_coop]

In the job, you would have the dependent library option, where you can mention the libraries you need installed. They will get installed on the job cluster before the job starts. This link might be useful...

[Tharun-Kumar · Accepted solution]

@erigaud 

You can traverse to the Tasks section of the job and to the first task in your job. Under the task properties, you would be seeing Dependent libraries using which you can install libraries from maven or python or even a custom JAR.

[erigaud]

Will installing a lib on a task for a job make this lib available for all the other tasks aswell ? (assuming they all use the same cluster)

[Tharun-Kumar]

@erigaud Yes it will be available for all the subsequent tasks on the cluster.

[MouhannadOweis]

How can you do this in ADF?

[daniel_sahal]

@MouhannadOweis 
You can also add libraries to ADF Databricks task.

[MouhannadOweis]

Thank you that worked.

[brushdoylt]

@daniel_sahalflappy bird

Is there another way to add libraries to the ADF Databricks task even faster?

[imsabarinath]

You may want to copy required libs to a volume and load it during cluster setup to avoid downloading the libs for every run.