- 3382 Views
- 3 replies
- 2 kudos
Passing values between notebook tasks in Workflow Jobs
I have created a Databricks workflow job with notebooks as individual tasks sequentially linked. I assign a value to a variable in one notebook task (ex: batchid = int(time.time()). Now, I want to pass this batchid variable to next notebook task.What...
- 3382 Views
- 3 replies
- 2 kudos
- 2 kudos
@brickster You would use dbutils.jobs.taskValues.set() and dbutils.jobs.taskValues.get().See docs for more details: https://docs.databricks.com/workflows/jobs/share-task-context.html
- 2 kudos
- 2788 Views
- 1 replies
- 0 kudos
Airflow - How to pull XComs value in the notebook task?
Using AIrflow, I have created a DAG with a sequence of notebook tasks. The first notebook returns a batch id; the subsequent notebook tasks need this batch_id.I am using the DatabricksSubmitRunOperator to run the notebook task. This operator pushes ...
- 2788 Views
- 1 replies
- 0 kudos
- 0 kudos
From what I understand - you want to pass a run_id parameter to the second notebook task?You can: Create a widget param inside your databricks notebook (https://docs.databricks.com/notebooks/widgets.html) that will consume your run_idPass the paramet...
- 0 kudos