DAB for_each_task - Passing task values

[DarioB]

I am trying to deploy a job with a for_each_task using DAB and Terraform and I am unable to properly pass the task value into the subsequent task.

These are my job tasks definition in the YAML:

      tasks:

        - task_key: FS_batching

          job_cluster_key: job_cluster_fs

          notebook_task:

            notebook_path: ...

        - task_key: FS_processing

          depends_on:

            - task_key: FS_batching

          for_each_task:

            inputs: {{tasks.FS_batching.values.var_name}}

            task:

              task_key: processing_iteration

              job_cluster_key: job_cluster_fs

              notebook_task:

                notebook_path: ....

                base_parameters:

                  input: {{input}}

For feeding the parameter into inputs and input fields I have tried different formats:

- {{tasks.FS_batching.values.var_name}}

- "{{tasks.FS_batching.values.var_name}}"

- ${tasks.FS_batching.values.var_name}

All of them failing with different errors. It seem that Terraform get crazy with those calls, any idea on how to solve it?

I am setting the parameter with: dbutils.jobs.taskValues.set(...) and retrieving it through: dbutils.widgets.get() because I am using scala in my notebooks and task values only seem to work with Python, so I use the widgets to retrieve them. But this should not mess with parameter setup into yaml.

[DarioB · Accepted solution]

We have been testing and find out the issue (I just realized that my anonymization of the names removed the source of the error).

We have tracked down to the inputs parameter of the for_each_task. It seems that is unable to call to task names with dashes: "{{tasks.Task-ETL_Get_Params.values.key}}" won't work because of the -. You can either use underscores or shortcut with ``: "{{tasks.`Task-ETL_Get_Params`.values.key}}".

I hope this helps to other people.