Databricks Jobs For Each Functionality

varshini_reddy — Wed, 04 Sep 2024 14:27:10 GMT

Hi Team, I have two questions, please clarigy.

1. can we use multiple tasks inside a for each for databricks jobs

2. is nested for each functionality available?

Re: Databricks Jobs For Each Functionality

szymon_dybczak — Wed, 04 Sep 2024 17:02:09 GMT

Hi @varshini_reddy ,

1. Currently you can only use one task inside foreach loop

2. No, currently you cannot nest foreach within another foreach

Re: Databricks Jobs For Each Functionality

filipniziol — Wed, 04 Sep 2024 19:47:44 GMT

Hi @varshini_reddy,

While running multiple tasks within a for each loop is not available, there is a simple workaround.

Workaround: create a main notebook to run multiple child notebooks:

Suppose you need to run notebook1, notebook2, and notebook3 within each iteration of the for each loop.
Instead of trying to include all three notebooks directly inside the for each, you can create a main notebook that sequentially runs all the required child notebooks.
The for each loop will then run this main notebook multiple times, which in turn will execute the child notebooks in sequence.

# main_notebook # Running the first child notebook dbutils.notebook.run("/path/to/notebook1", timeout_seconds=300) # Running the second child notebook dbutils.notebook.run("/path/to/notebook2", timeout_seconds=300) # Running the third child notebook dbutils.notebook.run("/path/to/notebook3", timeout_seconds=300)

topic Re: Databricks Jobs For Each Functionality in Data Engineering

Databricks Jobs For Each Functionality

Re: Databricks Jobs For Each Functionality

Re: Databricks Jobs For Each Functionality