topic Re: How do I pass arguments/variables from widgets to notebooks? in Data Engineering

How do I pass arguments/variables from widgets to notebooks?

lshar — Wed, 06 Apr 2022 15:42:23 GMT

Hello,

I am looking for a solution to this problem, which is known since 7 years: https://community.databricks.com/s/question/0D53f00001HKHZfCAP/how-do-i-pass-argumentsvariables-to-notebooks

What I need is to parametrize my notebooks using widget information. The parameters from the widgets should be passed to a %run of another notebook where all of the use case specific configurations are written.

Hence, I need solution to this problem, where the variable "john" is defined by a widget input.

----Cell1------------------------

john = 10

-----Cell2--------------------

%run path/to/NotebookB $VarA = john

submits "john" to NotebookB not the value of 10

I have found a workaround for this, hovewer the workaround is also having a problem.

Here is an example. As you can see in cmd 6 when I run the notebook with "%run" and a value the notebook is loaded.

In cmd5 I have tried the same, but inside an if-statement, which is failing with the explanation, that the file is missing. However, it is obvious, that the file exists.

Is anyone having the same issue and knows how to implement this?

Re: How do I pass arguments/variables from widgets to notebooks?

Hubert-Dudek — Wed, 06 Apr 2022 17:04:53 GMT

Can't you just use dbutils?

args = {}
args["VarA"] = dbutils.widgets.get("john")
dbutils.notebook.run("NotebookB", timeout=180, args)

Re: How do I pass arguments/variables from widgets to notebooks?

Ryan_Chynoweth — Wed, 06 Apr 2022 20:49:17 GMT

That is a great option to use notebook workflows. I don't think you can pass a variable into the %run commands. It needs to be a literal string.

%run typically cannot access variables/objects from different languages.

One thing to point out is that this method executes in a different session from the parent notebook. %run executes within the same session as the parent notebook.

Re: How do I pass arguments/variables from widgets to notebooks?

lshar — Thu, 28 Apr 2022 07:30:45 GMT

@Hubert Dudek @Kaniz Fatma

When I use the dbutils.notebook.run(..) a new cluster is started, hence I can run some other code, but cannot use variable and functions as if I have just run them directly in the same notebook. Hence, my goal is not met.

I want to run a function and use parameters from the notebook that is in the dbutils.notebook.run(), but this is not possible because of running with new cluster.

Re: How do I pass arguments/variables from widgets to notebooks?

Ryan_Chynoweth — Thu, 28 Apr 2022 16:15:44 GMT

Ishar, the dbutils.notebook.run() function is used to execute another notebook in a different session on the same cluster. Since the child notebook has a different session the variables, functions, parameters, classes, etc. are not available in the parent notebook.

If you wish to import a function from another notebook I would recommend using the %run functionality as that would execute the child notebook in the same session as the parent notebook.

To achieve your goal please use %run.

Additionally, if you are using python I would also look into our Python Import functionality that is available in Repos.

Re: How do I pass arguments/variables from widgets to notebooks?

lshar — Fri, 29 Apr 2022 06:58:06 GMT

Yes, exactly. This is what I am saying.

I use %run, however I want to pass a parameter, which is selected from the user (or given in a scheduled run) to have different configuration from the notebook, that I am running (example: I want two pass the parameter "female" and load other parameters and functions from the notebook, that I am running with %run, than if I pass "male")

Now the issue is with this parameter as I have already described in the first comment: I pass subtype, but when it is a parameter with value from a widget it is not working.

Re: How do I pass arguments/variables from widgets to notebooks?

Ryan_Chynoweth — Fri, 29 Apr 2022 17:58:25 GMT

Got it. So the only way to "pass a parameter" with %run is to define it as a variable in the parent notebook and use that variable in the child notebook. This works because both notebooks are executed in the same session so the variable my_var is available in both notebooks.

Parent Notebook:

my_var = "this is a parameter I want to pass" 
 
%run ./my_child_notebook

Child Notebook:

print(my_var) 
 
>> "this is a parameter I want to pass"

Re: How do I pass arguments/variables from widgets to notebooks?

T_Ash — Sun, 18 Aug 2024 18:06:20 GMT

Can we create paginated reports with multiple parameters(one parameter can dynamically change other parameter) or we can pass one variable from one dataset to other dataset like power bi paginated report using Databricks dashboard, please let me know..