Getting JOB-ID dynamically to create another job to refer as job-task

[ramy]

I am trying to create a new job in Databricks Asset Bundles which refers to another job-task and passing parameters to it. However, the previous job is not created yet (Or will be cretead using Databricks asset bundles in higher envs when deploying the bundle). I dont want to make it a hard-coded job-id.

[jeffrey597doss]

Hello!

To create a new job in Databricks Asset Bundles that refers to another job-task without hard-coding the job ID, use job parameters to pass dynamic values. Define a job parameter to hold the job ID once it’s created and use placeholders in your job configuration. This way, you can dynamically set the job ID when deploying the bundle.

[ramy]

Thanks for your response. However, it makes the process manual as when the first job is deployed, then get the first job-id and pass it to the second job to as a paramter-value and deploy the second job. I was looking for a way where the first job key (say name) can be used a reference parameter in the second job definition, so it stays dynamic.

[ally-r]

Good question. If the job is created outside of the asset bundle, and exists before this bundle will be used for the first time you can use a lookup variable

variables:
  your_job_id:
    lookup:
      job: "your_job_name"

If it's created inside the same bundle you can reference it directly

${resources.jobs.<job-name>.id}

I've not tried these in anger though, so let me know how you get on.

https://docs.databricks.com/en/dev-tools/bundles/variables.html

[nvashisth]

Hi Team,

Yes this solution of referencing the variable ${resources.jobs.<job-name>.id} works within the same bundle and this should be a solution. I had tried that by creating multiple workflows and referencing that.