Databricks Jobs For Each Functionality
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-04-2024 07:27 AM
Hi Team, I have two questions, please clarigy.
1. can we use multiple tasks inside a for each for databricks jobs
2. is nested for each functionality available?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-04-2024 10:02 AM
Hi @varshini_reddy ,
1. Currently you can only use one task inside foreach loop
2. No, currently you cannot nest foreach within another foreach
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-04-2024 12:47 PM
Hi @varshini_reddy,
While running multiple tasks within a for each loop is not available, there is a simple workaround.
Workaround: create a main notebook to run multiple child notebooks:
- Suppose you need to run notebook1, notebook2, and notebook3 within each iteration of the for each loop.
- Instead of trying to include all three notebooks directly inside the for each, you can create a main notebook that sequentially runs all the required child notebooks.
- The for each loop will then run this main notebook multiple times, which in turn will execute the child notebooks in sequence.
# main_notebook
# Running the first child notebook
dbutils.notebook.run("/path/to/notebook1", timeout_seconds=300)
# Running the second child notebook
dbutils.notebook.run("/path/to/notebook2", timeout_seconds=300)
# Running the third child notebook
dbutils.notebook.run("/path/to/notebook3", timeout_seconds=300)