- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-11-2023 12:09 AM
Simple question : what is the way to go to install libraries on job clusters ? There does not seem to be a "Libraries" tab on the UI as opposed to regular clusters. Does it mean that the only option is to use init scripts ?
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-11-2023 09:45 PM
You can traverse to the Tasks section of the job and to the first task in your job. Under the task properties, you would be seeing Dependent libraries using which you can install libraries from maven or python or even a custom JAR.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-11-2023 02:52 AM - edited 07-11-2023 02:54 AM
In the job, you would have the dependent library option, where you can mention the libraries you need installed. They will get installed on the job cluster before the job starts. This link might be useful...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-11-2023 09:45 PM
You can traverse to the Tasks section of the job and to the first task in your job. Under the task properties, you would be seeing Dependent libraries using which you can install libraries from maven or python or even a custom JAR.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-11-2023 11:00 PM
Will installing a lib on a task for a job make this lib available for all the other tasks aswell ? (assuming they all use the same cluster)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-12-2023 12:21 AM
@erigaud Yes it will be available for all the subsequent tasks on the cluster.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-24-2024 02:44 PM
How can you do this in ADF?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-24-2024 10:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-26-2024 08:54 AM
Thank you that worked.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-26-2024 12:31 AM
Is there another way to add libraries to the ADF Databricks task even faster?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-15-2024 12:31 AM
You may want to copy required libs to a volume and load it during cluster setup to avoid downloading the libs for every run.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
I am not able to select the requirements.txt file from my workspace folder, I can see the file but cannot select it. How do I overcome this problem?