Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-01-2021 02:35 AM
Notebook code is executed on driver to achieve parallelism you need just to create Spark dataframe with your list.
As @Werner Stinckens said you can run multiple notebooks together also so in that case you will not use list just to every notebook pass 1 parameter from your list:
from multiprocessing.pool import ThreadPool
my_params = ["Eu","JP","APAC"]
pool = ThreadPool(4) # match cpu cores here
pool.map(
lambda my_param: run_notebook("my_notebook", 3600, {"my_widget": my_param}),
my_params)
My blog: https://databrickster.medium.com/