Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-31-2025 09:31 AM
resources:
jobs:
my-first-job:
name: my-first-job
tasks:
- task_key: my-first-job-task
new_cluster:
spark_version: "13.3.x-scala2.12"
node_type_id: "i3.xlarge"
num_workers: 2
notebook_task:
notebook_path: ./src/test.py
my-second-job:
name: my-second-job
tasks:
- task_key: my-second-job-task
run_job_task:
job_id: ${resources.jobs.my-first-job.id}
In this example, the job_id of my-first-job is dynamically referenced in the run_job_task of my-second-job using ${resources.jobs.my-first-job.id}.
You could also use condition matching on the script https://docs.databricks.com/en/dev-tools/bundles/job-task-types.html#ifelse-condition-task