Dynamic Number of Tasks in Databricks Workflow

[FlexException]

Do Databricks workflows support creating a workflow with a dynamic number of tasks?

For example, let's say we have a DAG like this:

T1 ->    T2(1) ->

             T2(2) ->

              .....                 -> T3

             T2(n-1) ->

             T2(n) ->

In this case task 1 (T1) executes first and creates 1...n tasks (T2) that can execute in parallel. Then once all of those T2 tasks finish, a T3  task can run.

Here you don't know up front how many T2 tasks will exist up front because they rely on the output of T1 (which will change). In my specific case, T1 will generate n rows with each row providing the parameter needed to query a different server in T2.

[FlexException]

@Retired_modreally appreciate your reply.

I've looked through those blog posts and do not see any examples of achieving the dynamic task creation where I can retain the visibility you get through workflows across all the tasks.

Could you point me to an example?

All of the links that you posted lead to the same blog post from mid 2021, maybe I'm missing a key piece here.

Thanks!

[FlexException]

@Retired_modwould you happen to have more documentation on the process you outlined to achieve the dynamic task creation?

[YuliyanBogdanov]

Hey, @FlexException, 

Given you can create and update workflows dynamically using their JSON, it may turn out it is a matter of playing around with a JSON. See the blog below - let me know if it helps. Be cautious of how you would manage all of these dynamic changes though:

https://medium.com/@rishianand.nits/create-update-databricks-workflow-dynamically-abba4b0916b8

[tanyeesern]

@FlexException Databricks API supports job creation and execution Task Parameters and Values in Databricks Workflows | by Ryan Chynoweth | Medium
One possibility is after running earlier job, process the output to create a dynamic number of tasks in subsequent job