NandiniN
Databricks Employee
Databricks Employee
 resources:
  jobs:
    my-first-job:
      name: my-first-job
      tasks:
        - task_key: my-first-job-task
          new_cluster:
            spark_version: "13.3.x-scala2.12"
            node_type_id: "i3.xlarge"
            num_workers: 2
          notebook_task:
            notebook_path: ./src/test.py
    my-second-job:
      name: my-second-job
      tasks:
        - task_key: my-second-job-task
          run_job_task:
            job_id: ${resources.jobs.my-first-job.id}

In this example, the job_id of my-first-job is dynamically referenced in the run_job_task of my-second-job using ${resources.jobs.my-first-job.id}.

 

You could also use condition matching on the script https://docs.databricks.com/en/dev-tools/bundles/job-task-types.html#ifelse-condition-task

View solution in original post