topic Re: How to deploy an asset bundle job that triggers another one in Data Engineering

How to deploy an asset bundle job that triggers another one

dc-rnc — Thu, 30 Jan 2025 15:55:33 GMT

Hello everyone.

Using DAB, is there a dynamic value reference or something equivalent to get a job_id to be used inside the YAML definition of another Databricks job? I'd like to trigger that job from another one, but if I'm using a CI/CD pipeline to define/update the Databricks jobs in my workspace, that job_id is unknown at runtime.

For sure I can use some Databricks API and/or the Databricks CLI to get the job done (so, using a placeholder in the YAML file, identifying the correct job_id using API or CLI and then replacing it before deploying), but I was wondering if there is something more handy already.

Thank you. Cheers.

Re: How to deploy an asset bundle job that triggers another one

NandiniN — Fri, 31 Jan 2025 17:31:31 GMT

resources: jobs: my-first-job: name: my-first-job tasks: - task_key: my-first-job-task new_cluster: spark_version: "13.3.x-scala2.12" node_type_id: "i3.xlarge" num_workers: 2 notebook_task: notebook_path: ./src/test.py my-second-job: name: my-second-job tasks: - task_key: my-second-job-task run_job_task: job_id: ${resources.jobs.my-first-job.id}

In this example, the job_id of my-first-job is dynamically referenced in the run_job_task of my-second-job using ${resources.jobs.my-first-job.id}.

You could also use condition matching on the script https://docs.databricks.com/en/dev-tools/bundles/job-task-types.html#ifelse-condition-task