Managing libraries in workflows with multiple tasks - need to configure a list of libs for all tasks

[brian999]

I have workflows with multiple tasks, each of which need 5 different libraries to run. When I have to update those libraries, I have to go in and make the update in each and every task. So for one workflow I have 20 different places where I have to go through and update the libraries.

I need to be able to designate a list of libraries to be available on the job cluster for all the task that use it, so that I only have to update the libraries in one place.

But from what I can tell, an entirely new cluster definition gets created for job compute every time the workflow runs, so I don't have a single cluster to configure. What am I missing?

[daniel_sahal · Accepted solution]

@brian999 
Creating a requirements.txt would be a solution for you.

https://docs.databricks.com/en/libraries/workspace-files-libraries.html#add-dependent-libraries-to-w...

[brian999]

The libs I need to install are all private and not on Pypi. They are .whl files in repo folders. Can that be done with a requirements.txt file?

[daniel_sahal]

@brian999 

It should be doable.

In requirements.txt you can specify the path to the .whl file

[pablogarcia]

But it's requirement.txt always executed before we run a job in databricks? I have similar "issue" and the problem is that before the job execute the task it needs to have that dependency in the context, otherwise it wont be able to find the .py file 

[brian999 · Accepted solution]

Actually I think I found most of a solution here in one of the replies: https://community.databricks.com/t5/administration-architecture/installing-libraries-on-job-clusters...

It seems like I only have to define libs for the first task, and as long as all other tasks use the same job compute, I'm good to go. I'm assuming tasks within a workflow share compute by default?