topic Re: Managing libraries in workflows with multiple tasks - need to configure a list of libs for all t in Data Engineering

Managing libraries in workflows with multiple tasks - need to configure a list of libs for all tasks

brian999 — Wed, 10 Jul 2024 17:03:19 GMT

I have workflows with multiple tasks, each of which need 5 different libraries to run. When I have to update those libraries, I have to go in and make the update in each and every task. So for one workflow I have 20 different places where I have to go through and update the libraries.

I need to be able to designate a list of libraries to be available on the job cluster for all the task that use it, so that I only have to update the libraries in one place.

But from what I can tell, an entirely new cluster definition gets created for job compute every time the workflow runs, so I don't have a single cluster to configure. What am I missing?

Re: Managing libraries in workflows with multiple tasks - need to configure a list of libs for all t

daniel_sahal — Thu, 11 Jul 2024 07:01:56 GMT

@brian999
Creating a requirements.txt would be a solution for you.

https://docs.databricks.com/en/libraries/workspace-files-libraries.html#add-dependent-libraries-to-workflow-tasks-from-workspace-files

Re: Managing libraries in workflows with multiple tasks - need to configure a list of libs for all t

brian999 — Thu, 11 Jul 2024 12:34:05 GMT

The libs I need to install are all private and not on Pypi. They are .whl files in repo folders. Can that be done with a requirements.txt file?

Re: Managing libraries in workflows with multiple tasks - need to configure a list of libs for all t

brian999 — Thu, 11 Jul 2024 15:22:53 GMT

Actually I think I found most of a solution here in one of the replies: https://community.databricks.com/t5/administration-architecture/installing-libraries-on-job-clusters/m-p/37365/highlight/true#M245

It seems like I only have to define libs for the first task, and as long as all other tasks use the same job compute, I'm good to go. I'm assuming tasks within a workflow share compute by default?

Re: Managing libraries in workflows with multiple tasks - need to configure a list of libs for all t

daniel_sahal — Fri, 12 Jul 2024 05:29:52 GMT

@brian999

It should be doable.

In requirements.txt you can specify the path to the .whl file

Re: Managing libraries in workflows with multiple tasks - need to configure a list of libs for all t

pablogarcia — Tue, 02 Sep 2025 08:06:50 GMT

But it's requirement.txt always executed before we run a job in databricks? I have similar "issue" and the problem is that before the job execute the task it needs to have that dependency in the context, otherwise it wont be able to find the .py file