topic Re: Getting JOB-ID dynamically to create another job to refer as job-task in Data Engineering

Getting JOB-ID dynamically to create another job to refer as job-task

ramy — Fri, 06 Sep 2024 09:34:01 GMT

I am trying to create a new job in Databricks Asset Bundles which refers to another job-task and passing parameters to it. However, the previous job is not created yet (Or will be cretead using Databricks asset bundles in higher envs when deploying the bundle). I dont want to make it a hard-coded job-id.

Re: Getting JOB-ID dynamically to create another job to refer as job-task

jeffrey597doss — Fri, 06 Sep 2024 10:11:02 GMT

Hello!

To create a new job in Databricks Asset Bundles that refers to another job-task without hard-coding the job ID, use job parameters to pass dynamic values. Define a job parameter to hold the job ID once it’s created and use placeholders in your job configuration. This way, you can dynamically set the job ID when deploying the bundle.

Re: Getting JOB-ID dynamically to create another job to refer as job-task

ramy — Fri, 06 Sep 2024 10:25:38 GMT

Thanks for your response. However, it makes the process manual as when the first job is deployed, then get the first job-id and pass it to the second job to as a paramter-value and deploy the second job. I was looking for a way where the first job key (say name) can be used a reference parameter in the second job definition, so it stays dynamic.

Re: Getting JOB-ID dynamically to create another job to refer as job-task

ally-r — Fri, 06 Sep 2024 11:22:05 GMT

Good question. If the job is created outside of the asset bundle, and exists before this bundle will be used for the first time you can use a lookup variable

variables: your_job_id: lookup: job: "your_job_name"

If it's created inside the same bundle you can reference it directly

${resources.jobs.<job-name>.id}

I've not tried these in anger though, so let me know how you get on.

https://docs.databricks.com/en/dev-tools/bundles/variables.html

Re: Getting JOB-ID dynamically to create another job to refer as job-task

nvashisth — Fri, 20 Dec 2024 18:47:36 GMT

Hi Team,

Yes this solution of referencing the variable ${resources.jobs.<job-name>.id} works within the same bundle and this should be a solution. I had tried that by creating multiple workflows and referencing that.

Re: Getting JOB-ID dynamically to create another job to refer as job-task

priya12 — Wed, 12 Mar 2025 15:30:40 GMT

The lookup works. Here is how it can be used for a job existing outside the asset bundle

variables:
    my_jobid:
        description: Enter the Databricks Job name you want to refer.
        lookup:
          job: 'My Job1'

In the resources section, reference it like

tasks:
        - task_key: Run MyJob1
          run_job_task:
            job_id: ${var.my_jobid}