12-05-2023 06:16 AM
Do Databricks workflows support creating a workflow with a dynamic number of tasks?
For example, let's say we have a DAG like this:
T1 -> T2(1) ->
T2(2) ->
..... -> T3
T2(n-1) ->
T2(n) ->
In this case task 1 (T1) executes first and creates 1...n tasks (T2) that can execute in parallel. Then once all of those T2 tasks finish, a T3 task can run.
Here you don't know up front how many T2 tasks will exist up front because they rely on the output of T1 (which will change). In my specific case, T1 will generate n rows with each row providing the parameter needed to query a different server in T2.
12-06-2023 05:42 PM
@Retired_modreally appreciate your reply.
I've looked through those blog posts and do not see any examples of achieving the dynamic task creation where I can retain the visibility you get through workflows across all the tasks.
Could you point me to an example?
All of the links that you posted lead to the same blog post from mid 2021, maybe I'm missing a key piece here.
Thanks!
12-09-2023 05:00 AM
@Retired_modwould you happen to have more documentation on the process you outlined to achieve the dynamic task creation?
03-28-2024 11:19 PM
Hey, @FlexException,
Given you can create and update workflows dynamically using their JSON, it may turn out it is a matter of playing around with a JSON. See the blog below - let me know if it helps. Be cautious of how you would manage all of these dynamic changes though:
https://medium.com/@rishianand.nits/create-update-databricks-workflow-dynamically-abba4b0916b8
03-28-2024 08:46 PM
@FlexException Databricks API supports job creation and execution Task Parameters and Values in Databricks Workflows | by Ryan Chynoweth | Medium
One possibility is after running earlier job, process the output to create a dynamic number of tasks in subsequent job
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group