- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
Hello everyone.
Using DAB, is there a dynamic value reference or something equivalent to get a job_id to be used inside the YAML definition of another Databricks job? I'd like to trigger that job from another one, but if I'm using a CI/CD pipeline to define/update the Databricks jobs in my workspace, that job_id is unknown at runtime.
For sure I can use some Databricks API and/or the Databricks CLI to get the job done (so, using a placeholder in the YAML file, identifying the correct job_id using API or CLI and then replacing it before deploying), but I was wondering if there is something more handy already.
Thank you. Cheers.
- Labels:
-
Workflows
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
resources:
jobs:
my-first-job:
name: my-first-job
tasks:
- task_key: my-first-job-task
new_cluster:
spark_version: "13.3.x-scala2.12"
node_type_id: "i3.xlarge"
num_workers: 2
notebook_task:
notebook_path: ./src/test.py
my-second-job:
name: my-second-job
tasks:
- task_key: my-second-job-task
run_job_task:
job_id: ${resources.jobs.my-first-job.id}
In this example, the job_id
of my-first-job
is dynamically referenced in the run_job_task
of my-second-job
using ${resources.jobs.my-first-job.id}
.
You could also use condition matching on the script https://docs.databricks.com/en/dev-tools/bundles/job-task-types.html#ifelse-condition-task
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
resources:
jobs:
my-first-job:
name: my-first-job
tasks:
- task_key: my-first-job-task
new_cluster:
spark_version: "13.3.x-scala2.12"
node_type_id: "i3.xlarge"
num_workers: 2
notebook_task:
notebook_path: ./src/test.py
my-second-job:
name: my-second-job
tasks:
- task_key: my-second-job-task
run_job_task:
job_id: ${resources.jobs.my-first-job.id}
In this example, the job_id
of my-first-job
is dynamically referenced in the run_job_task
of my-second-job
using ${resources.jobs.my-first-job.id}
.
You could also use condition matching on the script https://docs.databricks.com/en/dev-tools/bundles/job-task-types.html#ifelse-condition-task

