topic Passing Parameters in a Workflow pipeline in Data Engineering

Passing Parameters in a Workflow pipeline

Flying_Rico — Fri, 13 Dec 2024 11:52:36 GMT

Hello Mates,

I’m currently working on four workflows, all of which are connected to my own notebook.
The four workflows should be started automatically one after the other and the only point that should be passed is the output of Workflow 1.
The workflow 1 reads a file, stores it in a table and edits the table name.
This table name should be accessed by Workflow 2 and thus get the table at the same time, so that Workflow 2 is then executed. With the same procedure, Workflow 3 and 4 should then run.
Currently I have set the job parameters Key = table_name and Value = {{tasks.File_Reader.values.table_name}} in workflow 2.
I want to set the task.value in Workflow 1 via dbutils.job.taskValue.set and then I want to get the table_name via the .get function.
Can someone suggest me a solution here, because I'm slowly running out of ideas on how to implement this?

Re: Passing Parameters in a Workflow pipeline

Walter_C — Fri, 13 Dec 2024 12:07:55 GMT

As part of the tasks in Workflow 1 you are setting a task with Run Job task type calling the Workflow 2? If yes you could send the parameter from there https://docs.databricks.com/en/jobs/run-job.html#:~:text=Use%20the%20Run%20Job%20task%20to%20trigger%20other%20jobs%20configured%20in%20your%20Databricks%20workspace.

Re: Passing Parameters in a Workflow pipeline

Flying_Rico — Fri, 13 Dec 2024 13:45:59 GMT

My problem is that the first task (workflow 1) works and also sets the parameter correctly when executed.
The second task can't find this parameter and can't access it accordingly, because apparently there are no available keys when checked.

Re: Passing Parameters in a Workflow pipeline

Walter_C — Fri, 13 Dec 2024 14:04:43 GMT

Yes the problem is that the .set and .get function works within the same job run, it does not pass to another job