topic Re: Pass variable from one notebook to another in Data Engineering

Pass variable from one notebook to another

ADB0513 — Tue, 06 Aug 2024 15:23:42 GMT

I have a main notebook where I am setting a python variable to the name of the catalog I want to work in. I then call another notebook, using %run, which runs an insert into using a SQL command where I want to specify the catalog using the catalog variable from the main notebook. What is the best way to accomplish this? Is there a way to pass the variable to the notebook that is being called from the %run command? Should I use a widget? Should I use spark.conf?

Thanks in advance for the help.

Re: Pass variable from one notebook to another

Retired_mod — Thu, 08 Aug 2024 08:07:09 GMT

Hi @ADB0513, To pass variables between notebooks in Databricks, you can use three main methods: **Widgets**, where you create and retrieve parameters using `dbutils.widgets` in both notebooks; **spark.conf**, where you set and get configuration parameters; and **Global Variables**, which are less recommended due to scope issues. Widgets and `spark.conf` are generally more robust and maintainable, especially for CI/CD setups.

Would you like more details on any of these methods? Or do you have any other questions about your CI/CD setup with Databricks Asset Bundles?

Re: Pass variable from one notebook to another

N_N — Tue, 11 Aug 2026 16:27:12 GMT

@Retired_mod

yes, it would be great to get more details on any of these methods. I am trying to do something similar too. Thank you.

Re: Pass variable from one notebook to another

adnan_alvee — Tue, 11 Aug 2026 17:52:31 GMT

Hi @ADB0513

With %run, you dont need to pass anything. %run runs the child inline in the same Python namespace, so the catalog variable set in the main notebook is already visible in the child. Just use it.

# main catalog = "dev_catalog" # %run ./child_notebook # child — catalog is already in scope; no widget, no config spark.sql(f"INSERT INTO `{catalog}`.target_schema.target_table " f"SELECT * FROM `{catalog}`.source_schema.source_table")

Regarding your other concerns:
- Do NOT use spark.conf for this because on serverless a custom key errors outright.
- Use a widget + dbutils.notebook.run() only if the child must be reusable / isolated (a real orchestration step), not for a plain %run helper.