cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to call a few child notebooks from master notebook parallelly?

andrew0117
Contributor

Planning using dbutils.notebook.run() to call all the child notebooks in the master notebook, but they are executed sequentially. 

1 ACCEPTED SOLUTION

Accepted Solutions

UmaMahesh1
Honored Contributor III

Hi @andrew liโ€‹ 

You can do this using scala or python constructs using threads and futures.

You can download and import the notebook archive from this link. It has the function to run notebooks parallelly.

https://docs.databricks.com/notebooks/notebook-workflows.html#run-multiple-notebooks-concurrently

After that, based on your preference, set the number of parallel notebooks to be run using numNotebooksInParallel variable in parallel-notebooks notebook .

Once done, you can call the parallelNotebooks function to run your notebooks parallelly. For examples on how to do that, refer Concurrent Notebooks notebook in that downloaded archive.

Be careful not to crash your driver by providing too many parallel notebooks.

Hope this helps.. Cheers.

Uma Mahesh D

View solution in original post

4 REPLIES 4

Anonymous
Not applicable

Hi @andrew liโ€‹ 

Great to meet you, and thanks for your question! 

Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon.

Thanks

UmaMahesh1
Honored Contributor III

Hi @andrew liโ€‹ 

You can do this using scala or python constructs using threads and futures.

You can download and import the notebook archive from this link. It has the function to run notebooks parallelly.

https://docs.databricks.com/notebooks/notebook-workflows.html#run-multiple-notebooks-concurrently

After that, based on your preference, set the number of parallel notebooks to be run using numNotebooksInParallel variable in parallel-notebooks notebook .

Once done, you can call the parallelNotebooks function to run your notebooks parallelly. For examples on how to do that, refer Concurrent Notebooks notebook in that downloaded archive.

Be careful not to crash your driver by providing too many parallel notebooks.

Hope this helps.. Cheers.

Uma Mahesh D

Thank you very much!

So, by using thread, all the jobs running child notebook are sharing the same cluster on which the master notebook is running?

UmaMahesh1
Honored Contributor III

Hi @andrew liโ€‹ 

Yes, They do run on the cluster on which the master notebook is running.

Specifically, we are multithreading the Spark driver with Futures to enable parallel job submission.

You can check out more on threads and futures for deeper understanding.

Hope this helps. Do mark the above as the best answer if it helped.

Cheers.

Uma Mahesh D

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group