Databricks Jobs For Each Functionality

varshini_reddy
New Contributor III

Hi Team, I have two questions, please clarigy.

1. can we use multiple tasks inside a for each for databricks jobs

2. is nested for each functionality available?

szymon_dybczak
Esteemed Contributor III

Hi @varshini_reddy , 

1. Currently you can only use one task inside foreach loop

2. No, currently you cannot nest foreach within another foreach

filipniziol
Esteemed Contributor

Hi @varshini_reddy,

While running multiple tasks within a for each loop is not available, there is a simple workaround.

Workaround: create a main notebook to run multiple child notebooks:

  • Suppose you need to run notebook1, notebook2, and notebook3 within each iteration of the for each loop.
  • Instead of trying to include all three notebooks directly inside the for each, you can create a main notebook that sequentially runs all the required child notebooks.
  • The for each loop will then run this main notebook multiple times, which in turn will execute the child notebooks in sequence.
# main_notebook

# Running the first child notebook
dbutils.notebook.run("/path/to/notebook1", timeout_seconds=300)

# Running the second child notebook
dbutils.notebook.run("/path/to/notebook2", timeout_seconds=300)

# Running the third child notebook
dbutils.notebook.run("/path/to/notebook3", timeout_seconds=300)