topic Re: Dynamic value input to a job in Data Engineering

Dynamic value input to a job

Hritik_Moon — Mon, 06 Oct 2025 06:39:22 GMT

How do I pass dynamic value to a databricks job?

I created a notebook which will extract the names of files in the catalog, I want to pass these names as parameter to another notebook task in a job.

What are the ways I can do this?

Re: Dynamic value input to a job

szymon_dybczak — Mon, 06 Oct 2025 06:48:17 GMT

Hi @Hritik_Moon ,

Define widgets in notebook, so you your notebooks are prepare for passing parameters. Then you can use task values to pass information between tasks. Check below documentation entry for more information and examples:

Use task values to pass information between tasks | Databricks on AWS

Re: Dynamic value input to a job

Hritik_Moon — Mon, 06 Oct 2025 07:07:07 GMT

Hello I tried this,

Notebook 1 :

dbutils.jobs.taskValues.set(key = "my_key", value = "hi From Notebook1")

Notebook2:

X = dbutils.jobs.taskValues.get(taskKey="01", key="my_key", debugValue = "Fail")

print(X)

Here I get "Fail" as output, its not fetching my_key

@szymon_dybczak

Re: Dynamic value input to a job

szymon_dybczak — Mon, 06 Oct 2025 07:53:35 GMT

Your task in the workflow is name 01?

Databricks widgets | Databricks on AWS

Re: Dynamic value input to a job

szymon_dybczak — Mon, 06 Oct 2025 08:13:23 GMT

Hi @Hritik_Moon ,

Folllow my steps. I created 2 notebooks:

- first one called Notebook1 with followign content

- second one called Notebook2 with following content that will read value defined in Notebook1

Here's my definiton of workflow that is using those 2 notebooks:

Pay attention to my taskKey in get method is named the same as the task in a workflow (Notebook1):

Re: Dynamic value input to a job

Hritik_Moon — Mon, 06 Oct 2025 09:47:10 GMT

Hello, I think you have posted wrong picture for notebook1. Could you please verify once.

I made some changes and its working now. 🙂

Thanks a lot.

Re: Dynamic value input to a job

szymon_dybczak — Mon, 06 Oct 2025 09:53:34 GMT

Hi @Hritik_Moon ,

Good catch, I've pasted wrong screenshot. Here's the correct one: