cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Databricks Jobs For Each Functionality

varshini_reddy
New Contributor III

Hi Team, I have two questions, please clarigy.

1. can we use multiple tasks inside a for each for databricks jobs

2. is nested for each functionality available?

2 REPLIES 2

szymon_dybczak
Contributor III

Hi @varshini_reddy , 

1. Currently you can only use one task inside foreach loop

2. No, currently you cannot nest foreach within another foreach

filipniziol
Contributor III

Hi @varshini_reddy,

While running multiple tasks within a for each loop is not available, there is a simple workaround.

Workaround: create a main notebook to run multiple child notebooks:

  • Suppose you need to run notebook1, notebook2, and notebook3 within each iteration of the for each loop.
  • Instead of trying to include all three notebooks directly inside the for each, you can create a main notebook that sequentially runs all the required child notebooks.
  • The for each loop will then run this main notebook multiple times, which in turn will execute the child notebooks in sequence.
# main_notebook

# Running the first child notebook
dbutils.notebook.run("/path/to/notebook1", timeout_seconds=300)

# Running the second child notebook
dbutils.notebook.run("/path/to/notebook2", timeout_seconds=300)

# Running the third child notebook
dbutils.notebook.run("/path/to/notebook3", timeout_seconds=300)

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group