topic Re: How to call a few child notebooks from master notebook parallelly? in Data Engineering

How to call a few child notebooks from master notebook parallelly?

andrew0117 — Sun, 27 Nov 2022 03:50:51 GMT

Planning using dbutils.notebook.run() to call all the child notebooks in the master notebook, but they are executed sequentially.

Re: How to call a few child notebooks from master notebook parallelly?

Anonymous — Sun, 27 Nov 2022 06:32:13 GMT

Hi @andrew li

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon.

Thanks

Re: How to call a few child notebooks from master notebook parallelly?

UmaMahesh1 — Sun, 27 Nov 2022 14:20:14 GMT

Hi @andrew li

You can do this using scala or python constructs using threads and futures.

You can download and import the notebook archive from this link. It has the function to run notebooks parallelly.

https://docs.databricks.com/notebooks/notebook-workflows.html#run-multiple-notebooks-concurrently

After that, based on your preference, set the number of parallel notebooks to be run using numNotebooksInParallel variable in parallel-notebooks notebook .

Once done, you can call the parallelNotebooks function to run your notebooks parallelly. For examples on how to do that, refer Concurrent Notebooks notebook in that downloaded archive.

Be careful not to crash your driver by providing too many parallel notebooks.

Hope this helps.. Cheers.

Re: How to call a few child notebooks from master notebook parallelly?

andrew0117 — Mon, 28 Nov 2022 21:37:40 GMT

Thank you very much!

So, by using thread, all the jobs running child notebook are sharing the same cluster on which the master notebook is running?

Re: How to call a few child notebooks from master notebook parallelly?

UmaMahesh1 — Tue, 29 Nov 2022 05:26:44 GMT

Hi @andrew li

Yes, They do run on the cluster on which the master notebook is running.

Specifically, we are multithreading the Spark driver with Futures to enable parallel job submission.

You can check out more on threads and futures for deeper understanding.

Hope this helps. Do mark the above as the best answer if it helped.

Cheers.