Pass variable from one notebook to another

ADB0513
Databricks Partner

‎08-06-2024 08:23 AM

I have a main notebook where I am setting a python variable to the name of the catalog I want to work in.  I then call another notebook, using %run, which runs an insert into using a SQL command where I want to specify the catalog using the catalog variable from the main notebook.  What is the best way to accomplish this?  Is there a way to pass the variable to the notebook that is being called from the %run command?  Should I use a widget?  Should I use spark.conf?

 

Thanks in advance for the help.

Retired_mod
Esteemed Contributor III

‎08-08-2024 01:07 AM

Hi @ADB0513, To pass variables between notebooks in Databricks, you can use three main methods: **Widgets**, where you create and retrieve parameters using `dbutils.widgets` in both notebooks; **spark.conf**, where you set and get configuration parameters; and **Global Variables**, which are less recommended due to scope issues. Widgets and `spark.conf` are generally more robust and maintainable, especially for CI/CD setups.

Would you like more details on any of these methods? Or do you have any other questions about your CI/CD setup with Databricks Asset Bundles?

N_N
New Contributor II

3 weeks ago - last edited 3 weeks ago

@Retired_mod 

yes, it would be great to get more details on any of these methods. I am trying to do something similar too. Thank you.

0 Kudos

adnan_alvee
Databricks Employee
Databricks Employee

3 weeks ago

Hi @ADB0513 

With %run, you dont need to pass anything. %run runs the child inline in the same Python namespace, so the catalog variable set in the main notebook is already visible in the child. Just use it.

# main
catalog = "dev_catalog"
# %run ./child_notebook

# child — catalog is already in scope; no widget, no config
spark.sql(f"INSERT INTO `{catalog}`.target_schema.target_table "
          f"SELECT * FROM `{catalog}`.source_schema.source_table")


Regarding your other concerns:
- Do NOT use spark.conf for this because on serverless a custom key errors outright. 
- Use a widget + dbutils.notebook.run() only if the child must be reusable / isolated (a real orchestration step), not for a plain %run helper.



0 Kudos