topic Re: Dynamic Number of Tasks in Databricks Workflow in Data Engineering

Dynamic Number of Tasks in Databricks Workflow

FlexException — Tue, 05 Dec 2023 14:16:13 GMT

Do Databricks workflows support creating a workflow with a dynamic number of tasks?

For example, let's say we have a DAG like this:

T1 -> T2(1) ->

T2(2) ->

..... -> T3

T2(n-1) ->

T2(n) ->

In this case task 1 (T1) executes first and creates 1...n tasks (T2) that can execute in parallel. Then once all of those T2 tasks finish, a T3 task can run.

Here you don't know up front how many T2 tasks will exist up front because they rely on the output of T1 (which will change). In my specific case, T1 will generate n rows with each row providing the parameter needed to query a different server in T2.

Re: Dynamic Number of Tasks in Databricks Workflow

FlexException — Thu, 07 Dec 2023 01:42:47 GMT

@Retired_modreally appreciate your reply.

I've looked through those blog posts and do not see any examples of achieving the dynamic task creation where I can retain the visibility you get through workflows across all the tasks.

Could you point me to an example?

All of the links that you posted lead to the same blog post from mid 2021, maybe I'm missing a key piece here.

Thanks!

Re: Dynamic Number of Tasks in Databricks Workflow

FlexException — Sat, 09 Dec 2023 13:00:37 GMT

@Retired_modwould you happen to have more documentation on the process you outlined to achieve the dynamic task creation?

Re: Dynamic Number of Tasks in Databricks Workflow

tanyeesern — Fri, 29 Mar 2024 03:46:23 GMT

@FlexException Databricks API supports job creation and execution Task Parameters and Values in Databricks Workflows | by Ryan Chynoweth | Medium
One possibility is after running earlier job, process the output to create a dynamic number of tasks in subsequent job

Re: Dynamic Number of Tasks in Databricks Workflow

YuliyanBogdanov — Fri, 29 Mar 2024 06:19:00 GMT

Hey, @FlexException,

Given you can create and update workflows dynamically using their JSON, it may turn out it is a matter of playing around with a JSON. See the blog below - let me know if it helps. Be cautious of how you would manage all of these dynamic changes though:

https://medium.com/@rishianand.nits/create-update-databricks-workflow-dynamically-abba4b0916b8