Requirement to run a databricks job from another job based on custom conditions using DAB

[adhi_databricks]

Hi everyone,

I'm using Databricks Asset Bundles to deploy a job that includes a run_job_task, which requires a job_id to trigger another job.

For different targets (dev, staging, prod), I need to pass different job_ids dynamically. To achieve this, I’ve defined the variable in the variables section and tried to override it per environment in the targets section.

However, when I run databricks bundle validate, I encounter the following error:

ERROR: Variable 'job_id' is incorrectly defined in target override

Here’s a simplified version of what I’m trying:

variables:
  job_id:
    description: Downstream job ID

targets:
  dev:
    variables:
      job_id: 1234

  staging:
    variables:
      job_id: 5678

  prod:
    variables:
      job_id: 91011

resources:
  jobs:
    main_job:
      name: trigger-${bundle.target}-downstream
      tasks:
        - task_key: trigger_downstream
          run_job_task:
            job_id: ${var.job_id}
            wait_for_completion: true

Is this the correct way to define and override variables for different targets in Asset Bundles? Or should this be handled differently (e.g., through variable-overrides.json or CLI)?

Appreciate any guidance!

Thanks in advance!

[-werners-]

can you try to add a default value for the variable definition (besides the description)? It can be anything.
For the rest this seems ok.

[adhi_databricks]

Have already tried adding the default value , still the same error was displayed.

[szymon_dybczak]

Hi @adhi_databricks ,

Could you also provide output of databricks bundle validate --debug ?

[szymon_dybczak]

@adhi_databricks  Also, one think that is always worth to do - update databricks cli to newest version. I tried to reproduce your error, but in my case everything went smoothly (without a need to define default variable in top-level mapping).

[-werners-]

ok, you do include the resources dir in your databricks.yml?
Because this should work.

[adhi_databricks]

Hey folks, Thanks for the help here
Was able to solve this issue with updating the databricks cli to latest version
Thanks once again!!

[szymon_dybczak · Accepted solution]

Hi @adhi_databricks ,

Glad that it work for you. If my answer was helpful, consider marking it as an accepted solution. This way, the next person with a similar question will be able to find the solution to their problem more quickly.