topic Re: Is it possible to passthrough job's parameters to variable? in Data Engineering

Is it possible to passthrough job's parameters to variable?

del1000 — Tue, 14 Sep 2021 15:19:28 GMT

Scenario:

I tried to run notebook_primary as a job with same parameters' map. This notebook is orchestrator for notebooks_sec_1, notebooks_sec_2, and notebooks_sec_3 and next. I run them by dbutils.notebook.run(path, timeout, arguments) function.

So how to get in notebook_primary all input parameters become from Job's configuration and pass them to notebooks_sec_... like i.e.:

arg = some_magic_function_gathering_all_actual_input_params()
 
#
# some iteration on arg
#
 
nb1 = dbutils.notebook.run('./notebooks_sec_1', 0, arg)
 
nb2= dbutils.notebook.run('./notebooks_sec_2', 0, arg)
 
nb3 = dbutils.notebook.run('./notebooks_sec_3', 0, arg)

Now I can't iterate through input params, I can get value when I know name of the parameter.

Thank you in advance for any advice.

Re: Is it possible to passthrough job's parameters to variable?

Dan_Z — Tue, 14 Sep 2021 18:12:15 GMT

Very possible. You just use dbutils.widgets.get().

For instance I set up a job with the following param:

{
  "foo": "bar"
}

The primary notebook:

the_arg = dbutils.widgets.get("foo")
 
print(the_arg)
 
nb1 = dbutils.notebook.run('./notebooks_sec_1', 0, {"foo" : the_arg})

notebooks_sec_1:

the_arg = dbutils.widgets.get("foo")
 
print(the_arg)

Then, when I ran it, both printed: "bar".

Re: Is it possible to passthrough job's parameters to variable?

del1000 — Tue, 14 Sep 2021 19:42:10 GMT

Sorry but it is not the answer for my question.

In notebook_primary, I don't know all names of node of arguments. I'd like to iterate arguments, do some change within it and send to process to each notebook_sec_* at all.

The question is: is the notebook able to know arguments which has been used to run it? Precisely, is it possible to write a below code?

for arg_key in arguments.keys():
 print(arg_key)

Re: Is it possible to passthrough job's parameters to variable?

Dan_Z — Tue, 14 Sep 2021 20:37:42 GMT

Oh, I see what you are looking for. Yes- totally possible. Here would be your primary notebook code:

all_args = dbutils.notebook.entry_point.getCurrentBindings()
 
print(all_args)
 
for arg in all_args:
  print(arg)
 
nb1 = dbutils.notebook.run('./notebooks_sec_1', 0, all_args)

Re: Is it possible to passthrough job's parameters to variable?

del1000 — Wed, 15 Sep 2021 09:03:05 GMT

Thank you, Dan. That is exactly what I wanted 🙂 By the way, where are these property and method of dbutils described? I can't find any references to them.

Re: Is it possible to passthrough job's parameters to variable?

673602 — Fri, 01 Oct 2021 06:51:22 GMT

We have exactly the same requirement but we were looking for a similar possibility using Scala code. We were not able to find any function close to getCurrentBindings() in scala.

Re: Is it possible to passthrough job's parameters to variable?

del1000 — Sun, 03 Oct 2021 18:56:09 GMT

@Balbir Singh , I'm newbie in Databricks but the manual says you can use a python cell and transfer variables to scala's cell by temp tables.

https://docs.databricks.com/notebooks/notebook-workflows.html#pass-structured-data

Re: Is it possible to passthrough job's parameters to variable?

nnalla — Mon, 16 Sep 2024 23:32:55 GMT

Re: Is it possible to passthrough job's parameters to variable?

nnalla — Mon, 16 Sep 2024 23:36:57 GMT

I am using getCurrentBindings(), but it returns an empty dictionary even though I passed parameters. I am running it in a scheduled workflow job