cancel
Showing results forĀ 
Search instead forĀ 
Did you mean:Ā 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forĀ 
Search instead forĀ 
Did you mean:Ā 

Passing Parameters in a Workflow pipeline

Flying_Rico
New Contributor II

Hello Mates,

Iā€™m currently working on four workflows, all of which are connected to my own notebook.
The four workflows should be started automatically one after the other and the only point that should be passed is the output of Workflow 1.
The workflow 1 reads a file, stores it in a table and edits the table name.
This table name should be accessed by Workflow 2 and thus get the table at the same time, so that Workflow 2 is then executed. With the same procedure, Workflow 3 and 4 should then run.
Currently I have set the job parameters Key = table_name and Value = {{tasks.File_Reader.values.table_name}} in workflow 2.
I want to set the task.value in Workflow 1 via dbutils.job.taskValue.set and then I want to get the table_name via the .get function.
Can someone suggest me a solution here, because I'm slowly running out of ideas on how to implement this?

3 REPLIES 3

Walter_C
Databricks Employee
Databricks Employee

As part of the tasks in Workflow 1 you are setting a task with Run Job task type calling the Workflow 2? If yes you could send the parameter from there https://docs.databricks.com/en/jobs/run-job.html#:~:text=Use%20the%20Run%20Job%20task%20to%20trigger...

Flying_Rico
New Contributor II

My problem is that the first task (workflow 1) works and also sets the parameter correctly when executed.
The second task can't find this parameter and can't access it accordingly, because apparently there are no available keys when checked.

Walter_C
Databricks Employee
Databricks Employee

Yes the problem is that the .set and .get function works within the same job run, it does not pass to another job

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonā€™t want to miss the chance to attend and share knowledge.

If there isnā€™t a group near you, start one and help create a community that brings people together.

Request a New Group