- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-20-2024 09:53 AM
To pass multiple values into the job that is referenced in Task_B, you can use dynamic value references. Dynamic value references allow you to reference task values set in upstream tasks. This is a recommended approach by Databricks as it can be used with multiple task types.
In your case, you can set the values in Task_A using dbutils.jobs.taskValues.set(). For example, if you want to set multiple entity_ids, you can do:
python
dbutils.jobs.taskValues.set(key = "entity_ids", value = [1, 2])
Then, in Task_B, you can reference these values using dynamic value references. The syntax for this is {{tasks.Task_A.values.entity_ids}}.
Please note that the dynamic value references are used in the job settings, not in the notebook code.This is how you would set it in the job settings:
json
"parameters": {
"entity_ids": "{{tasks.Task_A.values.entity_ids}}"
}
This way, the entity_ids set in Task_A will be passed to Task_B.