https://community.databricks.com/t5/data-engineering/databricks-workflows-generate-tasks-programmatically/m-p/99808#M40104 <P data-unlink="true">This sounds like a great fit for the For Each task type! <A href="https://www.databricks.com/blog/streamlining-repetitive-tasks-databricks-workflows" target="_self">Here is the blog</A>, and the <A href="https://docs.databricks.com/en/jobs/for-each.html" target="_self">documentation</A></P> Fri, 22 Nov 2024 18:51:05 GMT cgrant 2024-11-22T18:51:05Z https://community.databricks.com/t5/data-engineering/databricks-workflows-generate-tasks-programmatically/m-p/99633#M40052 <P>Hi there,</P><P>I've used databricks workflows to explicitly create tasks with known input parameters (either user input or default parameters). But I'm wondering, what if I want the output of one task to be a list of specific ID's (e.g. id = [7,8,10,13,27]), and then I want individual parallel tasks generated having these ID's be the new input parameters for downstream tasks.</P><P>Is this possible to implement out of the box with default Databricks Workflows (GUI)? Or for this example, would I need to leverage something like Databricks Jobs API to generate workflow tasks programmatically as per my example?</P> Thu, 21 Nov 2024 14:31:45 GMT https://community.databricks.com/t5/data-engineering/databricks-workflows-generate-tasks-programmatically/m-p/99633#M40052 ChristianRRL 2024-11-21T14:31:45Z https://community.databricks.com/t5/data-engineering/databricks-workflows-generate-tasks-programmatically/m-p/99808#M40104 <P data-unlink="true">This sounds like a great fit for the For Each task type! <A href="https://www.databricks.com/blog/streamlining-repetitive-tasks-databricks-workflows" target="_self">Here is the blog</A>, and the <A href="https://docs.databricks.com/en/jobs/for-each.html" target="_self">documentation</A></P> Fri, 22 Nov 2024 18:51:05 GMT https://community.databricks.com/t5/data-engineering/databricks-workflows-generate-tasks-programmatically/m-p/99808#M40104 cgrant 2024-11-22T18:51:05Z