@MihaiTache You can achieve this by utilizing a combination of
dbutils.widgets.get and dbutils.jobs.taskValues.set. The approach involves extracting the run_id from Job1 and passing it as a value to Job2 using taskValues.set. This allows seamless communication between jobs within a workflow, enabling Job2 to access information or outputs from Job1 efficiently.
In Job1:
Extract necessary values (e.g., run_id) using dbutils.widgets.get to retrieve any required parameters passed to the job. Use dbutils.jobs.taskValues.set to store and make the value available for subsequent jobs.
In Job2:
Retrieve the value set by Job1 using dbutils.jobs.taskValues.get. This ensures Job2 starts with the required context, such as Job1โs run_id or other task-related values.
Example:
Job1 & Job 2 Setup screenshots: Review the attachment
Job1 Notebook:
job1_run_id = dbutils.widgets.get("job1_run_id")
dbutils.jobs.taskValues.set("job1_run_id", job1_run_id)
Job2 Notebook:
extract_job1_run_id = dbutils.widgets.get("extract_job1_run_id")
print(f"Received extract_job1_run_id: {extract_job1_run_id}")