How to deploy an asset bundle job that triggers another one

dc-rnc
Contributor

‎01-30-2025 07:55 AM

Hello everyone.

Using DAB, is there a dynamic value reference or something equivalent to get a job_id to be used inside the YAML definition of another Databricks job? I'd like to trigger that job from another one, but if I'm using a CI/CD pipeline to define/update the Databricks jobs in my workspace, that job_id is unknown at runtime.

For sure I can use some Databricks API and/or the Databricks CLI to get the job done (so, using a placeholder in the YAML file, identifying the correct job_id using API or CLI and then replacing it before deploying), but I was wondering if there is something more handy already.

Thank you. Cheers.

Labels:
0 Kudos

NandiniN
Databricks Employee
Databricks Employee

‎01-31-2025 09:31 AM 

 resources:
  jobs:
    my-first-job:
      name: my-first-job
      tasks:
        - task_key: my-first-job-task
          new_cluster:
            spark_version: "13.3.x-scala2.12"
            node_type_id: "i3.xlarge"
            num_workers: 2
          notebook_task:
            notebook_path: ./src/test.py
    my-second-job:
      name: my-second-job
      tasks:
        - task_key: my-second-job-task
          run_job_task:
            job_id: ${resources.jobs.my-first-job.id}

In this example, the job_id of my-first-job is dynamically referenced in the run_job_task of my-second-job using ${resources.jobs.my-first-job.id}.

 

You could also use condition matching on the script https://docs.databricks.com/en/dev-tools/bundles/job-task-types.html#ifelse-condition-task

View solution in original post