cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to pass dynamic value in databricks

Siddhesh2525
New Contributor III

I have separate column value defined in 13 diffrent notebook and i want merge into 1 databrick notebook and want to pass dynamic parameter using databrick so it will help me to run in single databricks notebook .

2 REPLIES 2

Hubert-Dudek
Esteemed Contributor III

Hi you can run one notebook from another and pass value as widget:

run_notebook(notebook, timeout, widgets_value)

Here is more info https://docs.databricks.com/notebooks/notebook-workflows.html

You can even run them in many processes simultaneously:

    from multiprocessing.pool import ThreadPool
    my_params =  ["Eu","JP","APAC"]
    pool = ThreadPool(4) # match cpu cores here
    pool.map(
        lambda my_param: run_notebook("my_notebook", 3600, {"my_widget": my_param}),
        my_params)

Prabakar
Esteemed Contributor III
Esteemed Contributor III

Hi @siddhesh Bhavar​ you can use widgets with the %run command to achieve this.

https://docs.databricks.com/notebooks/widgets.html#use-widgets-with-run

%run /path/to/notebook $X="10" $Y="1"

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.