How to call a few child notebooks from master notebook parallelly?

[andrew0117]

Planning using dbutils.notebook.run() to call all the child notebooks in the master notebook, but they are executed sequentially. 

[Anonymous]

Hi @andrew li​ 

Great to meet you, and thanks for your question! 

Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon.

Thanks

[UmaMahesh1 · Accepted solution]

Hi @andrew li​ 

You can do this using scala or python constructs using threads and futures.

You can download and import the notebook archive from this link. It has the function to run notebooks parallelly.

https://docs.databricks.com/notebooks/notebook-workflows.html#run-multiple-notebooks-concurrently

After that, based on your preference, set the number of parallel notebooks to be run using numNotebooksInParallel variable in parallel-notebooks notebook .

Once done, you can call the parallelNotebooks function to run your notebooks parallelly. For examples on how to do that, refer Concurrent Notebooks notebook in that downloaded archive.

Be careful not to crash your driver by providing too many parallel notebooks.

Hope this helps.. Cheers.

[andrew0117]

Thank you very much!

So, by using thread, all the jobs running child notebook are sharing the same cluster on which the master notebook is running?

[UmaMahesh1]

Hi @andrew li​ 

Yes, They do run on the cluster on which the master notebook is running.

Specifically, we are multithreading the Spark driver with Futures to enable parallel job submission.

You can check out more on threads and futures for deeper understanding.

Hope this helps. Do mark the above as the best answer if it helped.

Cheers.