โ07-11-2023 12:09 AM
Simple question : what is the way to go to install libraries on job clusters ? There does not seem to be a "Libraries" tab on the UI as opposed to regular clusters. Does it mean that the only option is to use init scripts ?
โ07-11-2023 09:45 PM
You can traverse to the Tasks section of the job and to the first task in your job. Under the task properties, you would be seeing Dependent libraries using which you can install libraries from maven or python or even a custom JAR.
โ07-11-2023 02:52 AM - edited โ07-11-2023 02:54 AM
In the job, you would have the dependent library option, where you can mention the libraries you need installed. They will get installed on the job cluster before the job starts. This link might be useful...
โ07-11-2023 09:45 PM
You can traverse to the Tasks section of the job and to the first task in your job. Under the task properties, you would be seeing Dependent libraries using which you can install libraries from maven or python or even a custom JAR.
โ07-11-2023 11:00 PM
Will installing a lib on a task for a job make this lib available for all the other tasks aswell ? (assuming they all use the same cluster)
โ07-12-2023 12:21 AM
@erigaud Yes it will be available for all the subsequent tasks on the cluster.
a month ago
How can you do this in ADF?
a month ago
4 weeks ago
Thank you that worked.
4 weeks ago
Is there another way to add libraries to the ADF Databricks task even faster?