โ07-11-2023 12:09 AM
Simple question : what is the way to go to install libraries on job clusters ? There does not seem to be a "Libraries" tab on the UI as opposed to regular clusters. Does it mean that the only option is to use init scripts ?
โ07-11-2023 09:45 PM
You can traverse to the Tasks section of the job and to the first task in your job. Under the task properties, you would be seeing Dependent libraries using which you can install libraries from maven or python or even a custom JAR.
โ07-11-2023 02:52 AM - edited โ07-11-2023 02:54 AM
In the job, you would have the dependent library option, where you can mention the libraries you need installed. They will get installed on the job cluster before the job starts. This link might be useful...
โ07-11-2023 09:45 PM
You can traverse to the Tasks section of the job and to the first task in your job. Under the task properties, you would be seeing Dependent libraries using which you can install libraries from maven or python or even a custom JAR.
โ07-11-2023 11:00 PM
Will installing a lib on a task for a job make this lib available for all the other tasks aswell ? (assuming they all use the same cluster)
โ07-12-2023 12:21 AM
@erigaud Yes it will be available for all the subsequent tasks on the cluster.
a week ago
How can you do this in ADF?
a week ago
a week ago
Thank you that worked.
a week ago
Is there another way to add libraries to the ADF Databricks task even faster?
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.