โ11-26-2022 07:50 PM
Planning using dbutils.notebook.run() to call all the child notebooks in the master notebook, but they are executed sequentially.
โ11-27-2022 06:20 AM
Hi @andrew liโ
You can do this using scala or python constructs using threads and futures.
You can download and import the notebook archive from this link. It has the function to run notebooks parallelly.
https://docs.databricks.com/notebooks/notebook-workflows.html#run-multiple-notebooks-concurrently
After that, based on your preference, set the number of parallel notebooks to be run using numNotebooksInParallel variable in parallel-notebooks notebook .
Once done, you can call the parallelNotebooks function to run your notebooks parallelly. For examples on how to do that, refer Concurrent Notebooks notebook in that downloaded archive.
Be careful not to crash your driver by providing too many parallel notebooks.
Hope this helps.. Cheers.
โ11-26-2022 10:32 PM
Hi @andrew liโ
Great to meet you, and thanks for your question!
Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon.
Thanks
โ11-27-2022 06:20 AM
Hi @andrew liโ
You can do this using scala or python constructs using threads and futures.
You can download and import the notebook archive from this link. It has the function to run notebooks parallelly.
https://docs.databricks.com/notebooks/notebook-workflows.html#run-multiple-notebooks-concurrently
After that, based on your preference, set the number of parallel notebooks to be run using numNotebooksInParallel variable in parallel-notebooks notebook .
Once done, you can call the parallelNotebooks function to run your notebooks parallelly. For examples on how to do that, refer Concurrent Notebooks notebook in that downloaded archive.
Be careful not to crash your driver by providing too many parallel notebooks.
Hope this helps.. Cheers.
โ11-28-2022 01:37 PM
Thank you very much!
So, by using thread, all the jobs running child notebook are sharing the same cluster on which the master notebook is running?
โ11-28-2022 09:26 PM
Hi @andrew liโ
Yes, They do run on the cluster on which the master notebook is running.
Specifically, we are multithreading the Spark driver with Futures to enable parallel job submission.
You can check out more on threads and futures for deeper understanding.
Hope this helps. Do mark the above as the best answer if it helped.
Cheers.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group