Passing Parameters in a Workflow pipeline
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ā12-13-2024 03:52 AM
Hello Mates,
Iām currently working on four workflows, all of which are connected to my own notebook.
The four workflows should be started automatically one after the other and the only point that should be passed is the output of Workflow 1.
The workflow 1 reads a file, stores it in a table and edits the table name.
This table name should be accessed by Workflow 2 and thus get the table at the same time, so that Workflow 2 is then executed. With the same procedure, Workflow 3 and 4 should then run.
Currently I have set the job parameters Key = table_name and Value = {{tasks.File_Reader.values.table_name}} in workflow 2.
I want to set the task.value in Workflow 1 via dbutils.job.taskValue.set and then I want to get the table_name via the .get function.
Can someone suggest me a solution here, because I'm slowly running out of ideas on how to implement this?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ā12-13-2024 04:07 AM
As part of the tasks in Workflow 1 you are setting a task with Run Job task type calling the Workflow 2? If yes you could send the parameter from there https://docs.databricks.com/en/jobs/run-job.html#:~:text=Use%20the%20Run%20Job%20task%20to%20trigger....
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ā12-13-2024 05:45 AM
My problem is that the first task (workflow 1) works and also sets the parameter correctly when executed.
The second task can't find this parameter and can't access it accordingly, because apparently there are no available keys when checked.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ā12-13-2024 06:04 AM
Yes the problem is that the .set and .get function works within the same job run, it does not pass to another job

