DAB for_each_task - Passing task values

DarioB — Thu, 05 Jun 2025 16:23:52 GMT

I am trying to deploy a job with a for_each_task using DAB and Terraform and I am unable to properly pass the task value into the subsequent task.

These are my job tasks definition in the YAML:

tasks:

- task_key: FS_batching

job_cluster_key: job_cluster_fs

notebook_task:

notebook_path: ...

- task_key: FS_processing

depends_on:

- task_key: FS_batching

for_each_task:

inputs: {{tasks.FS_batching.values.var_name}}

task:

task_key: processing_iteration

job_cluster_key: job_cluster_fs

notebook_task:

notebook_path: ....

base_parameters:

input: {{input}}

For feeding the parameter into inputs and input fields I have tried different formats:

- {{tasks.FS_batching.values.var_name}}

- "{{tasks.FS_batching.values.var_name}}"

- ${tasks.FS_batching.values.var_name}

All of them failing with different errors. It seem that Terraform get crazy with those calls, any idea on how to solve it?

I am setting the parameter with: dbutils.jobs.taskValues.set(...) and retrieving it through: dbutils.widgets.get() because I am using scala in my notebooks and task values only seem to work with Python, so I use the widgets to retrieve them. But this should not mess with parameter setup into yaml.

Re: DAB for_each_task - Passing task values

DarioB — Tue, 10 Jun 2025 14:35:27 GMT

We have been testing and find out the issue (I just realized that my anonymization of the names removed the source of the error).

We have tracked down to the inputs parameter of the for_each_task. It seems that is unable to call to task names with dashes: "{{tasks.Task-ETL_Get_Params.values.key}}" won't work because of the -. You can either use underscores or shortcut with ``: "{{tasks.`Task-ETL_Get_Params`.values.key}}".

I hope this helps to other people.

topic Re: DAB for_each_task - Passing task values in Data Engineering

DAB for_each_task - Passing task values

Re: DAB for_each_task - Passing task values