Hi,
I would like to use the new "Job as Task" feature but Im having trouble to pass values.
Scenario
I have a workflow job which contains 2 tasks.
- Task_A (type "Notebook"): Read data from a table and based on the contents decide, whether the workflow in Task_B should be executed (or not).
- Task_B (type "Run Job"): This references another Workflow that itself consists of multiple tasks, all of type "Notebook". The workflow takes several parameters. For the sake of brevity here, let's just assume a parameter "entity_ids".
The referenced workflow from Task_B runs totally fine on it's own when started manually.
For this scenario I would like to add some logic to automate execution of that workflow.
Based on what Task_A decides, I would like to start Task_B with the appropriate list of "entity_id" parameters values.
I had tried to set this with dbutils.jobs.taskValues.set("entity_id", "[1, 2]") in Task_A and read with dbutils.jobs.taskValues.get("Task_A", "entity_ids", debugValue="[]" ) in the first Notebook of Task_B, but this throws an error within the nested job:
Task key does not exist in run: Task_A.
I guess that the workflow that is referenced in Task_B is unaware of the parent workflow and might be run in a different context, and therefore cannot find the taskKey == "Task_A".
To verify my assumption, I tried changing the type of Task_B into a Notebook. Reading the value now works fine.
My question:
How do I pass (multiple) values into the job that is referenced in Task_B?
Thank you for your help!