How to pass dynamic value in databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-01-2021 03:16 AM
I have separate column value defined in 13 diffrent notebook and i want merge into 1 databrick notebook and want to pass dynamic parameter using databrick so it will help me to run in single databricks notebook .
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-01-2021 03:32 AM
Hi you can run one notebook from another and pass value as widget:
run_notebook(notebook, timeout, widgets_value)
Here is more info https://docs.databricks.com/notebooks/notebook-workflows.html
You can even run them in many processes simultaneously:
from multiprocessing.pool import ThreadPool
my_params = ["Eu","JP","APAC"]
pool = ThreadPool(4) # match cpu cores here
pool.map(
lambda my_param: run_notebook("my_notebook", 3600, {"my_widget": my_param}),
my_params)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-02-2021 07:06 AM
Hi @siddhesh Bhavar you can use widgets with the %run command to achieve this.
https://docs.databricks.com/notebooks/widgets.html#use-widgets-with-run
%run /path/to/notebook $X="10" $Y="1"

