topic Re: Requirement to run a databricks job from another job based on custom conditions using DAB in Data Engineering

Requirement to run a databricks job from another job based on custom conditions using DAB

adhi_databricks — Mon, 14 Jul 2025 08:59:03 GMT

Hi everyone,

I'm using Databricks Asset Bundles to deploy a job that includes a run_job_task, which requires a job_id to trigger another job.

For different targets (dev, staging, prod), I need to pass different job_ids dynamically. To achieve this, I’ve defined the variable in the variables section and tried to override it per environment in the targets section.

However, when I run databricks bundle validate, I encounter the following error:

ERROR: Variable 'job_id' is incorrectly defined in target override

Here’s a simplified version of what I’m trying:

variables: job_id: description: Downstream job ID targets: dev: variables: job_id: 1234 staging: variables: job_id: 5678 prod: variables: job_id: 91011 resources: jobs: main_job: name: trigger-${bundle.target}-downstream tasks: - task_key: trigger_downstream run_job_task: job_id: ${var.job_id} wait_for_completion: true

Is this the correct way to define and override variables for different targets in Asset Bundles? Or should this be handled differently (e.g., through variable-overrides.json or CLI)?

Appreciate any guidance!

Thanks in advance!

Re: Requirement to run a databricks job from another job based on custom conditions using DAB

-werners- — Mon, 14 Jul 2025 09:30:41 GMT

can you try to add a default value for the variable definition (besides the description)? It can be anything.
For the rest this seems ok.

Re: Requirement to run a databricks job from another job based on custom conditions using DAB

adhi_databricks — Mon, 14 Jul 2025 09:58:35 GMT

Have already tried adding the default value , still the same error was displayed.

Re: Requirement to run a databricks job from another job based on custom conditions using DAB

-werners- — Mon, 14 Jul 2025 10:08:35 GMT

ok, you do include the resources dir in your databricks.yml?
Because this should work.

Re: Requirement to run a databricks job from another job based on custom conditions using DAB

szymon_dybczak — Mon, 14 Jul 2025 10:19:11 GMT

Hi @adhi_databricks ,

Could you also provide output of databricks bundle validate --debug ?

Re: Requirement to run a databricks job from another job based on custom conditions using DAB

szymon_dybczak — Mon, 14 Jul 2025 10:31:16 GMT

@adhi_databricks Also, one think that is always worth to do - update databricks cli to newest version. I tried to reproduce your error, but in my case everything went smoothly (without a need to define default variable in top-level mapping).

Re: Requirement to run a databricks job from another job based on custom conditions using DAB

adhi_databricks — Wed, 16 Jul 2025 05:45:59 GMT

Hey folks, Thanks for the help here
Was able to solve this issue with updating the databricks cli to latest version
Thanks once again!!

Re: Requirement to run a databricks job from another job based on custom conditions using DAB

szymon_dybczak — Wed, 16 Jul 2025 06:39:53 GMT

Hi @adhi_databricks ,

Glad that it work for you. If my answer was helpful, consider marking it as an accepted solution. This way, the next person with a similar question will be able to find the solution to their problem more quickly.