Databricks Jobs For Each Functionality

[varshini_reddy]

Hi Team, I have two questions, please clarigy.

1. can we use multiple tasks inside a for each for databricks jobs

2. is nested for each functionality available?

[szymon_dybczak]

Hi @varshini_reddy , 

1. Currently you can only use one task inside foreach loop

2. No, currently you cannot nest foreach within another foreach

[filipniziol]

Hi @varshini_reddy,

While running multiple tasks within a for each loop is not available, there is a simple workaround.

Workaround: create a main notebook to run multiple child notebooks:

• Suppose you need to run notebook1, notebook2, and notebook3 within each iteration of the for each loop.
• Instead of trying to include all three notebooks directly inside the for each, you can create a main notebook that sequentially runs all the required child notebooks.
• The for each loop will then run this main notebook multiple times, which in turn will execute the child notebooks in sequence.

# main_notebook

# Running the first child notebook
dbutils.notebook.run("/path/to/notebook1", timeout_seconds=300)

# Running the second child notebook
dbutils.notebook.run("/path/to/notebook2", timeout_seconds=300)

# Running the third child notebook
dbutils.notebook.run("/path/to/notebook3", timeout_seconds=300)