Re: If statement in DAB YAML file support

QuanDo1 · Monday

Hi,
Is there a plan to have if statement support in DAB YAML file?

For example, I would like to have different schedule for jobs based on the environment (higher frequency in PROD, lower frequency in DEV, QA). How can I do it in Databricks via workflow yml files?

balajij8 · Monday

Hi QuanDo1,

DABs handle environment-specific schedules natively through declarative target variable overrides instead of relying on procedural if statements. You define top-level variables in yml (job_schedule_cron,job_schedule_timezone) and assign baseline defaults that act as fallbacks. Under each target definition (dev, qa, prod), you override these values according to environment requirements - setting job_schedule_cron to a weekly cron expression for dev, daily for qa and every two hours for prod. In the job definition under resources/jobs.yml, you can reference the variable directly as var.job_schedule_cron allowing DABs to resolve and substitute the schedule dynamically from variables.

targets:
  dev:
    default: true
    mode: development
    variables:
      job_schedule_cron: "0 0 9 ? * MON"  # Weekly on Mondays at 9am
      environment: "dev"
  
  qa:
    mode: development
    variables:
      job_schedule_cron: "0 0 6 * * ?"  # Daily at 6am
      environment: "qa"

When executing bundles the CLI reads the relevant dev/qa/prod substitutes the cron expression wherever var.job_schedule_cron is referenced in the resources, and provisions a job scheduled based on it. It provides type-safety through default values, keeps schedule differences transparent during code reviews and scales cleanly to parameterize other scheduling attributes such as execution timezones or the pause_status toggle without introducing conditional template logic.

Niyojit · Monday

Hi @QuanDo1

You don't need to use if/else for this use case. Databricks Asset Bundles (DAB) already supports variables that can be overridden per target environment.

You can define your variables once and then assign different values for each target (dev, qa, prod). This keeps your bundle clean and avoids conditional logic in the YAML.

bundle:
  name: dynamic-scheduler-demo

variables:
  job_cron_schedule:
    description: "The cron expression for running the workflow"
    default: "0 0 12 * * ?"

  job_timezone:
    description: "The timezone for the schedule"
    default: "UTC"

targets:
  dev:
    mode: development
    default: true
    variables:
      job_cron_schedule: "0 0/15 * * * ?"  # Every 15 minutes

resources:
  jobs:
    my_scheduled_job:
      name: "My Scheduled Data Pipeline"
      schedule:
        quartz_cron_expression: "${var.job_cron_schedule}"
        timezone_id: "${var.job_timezone}"
        pause_status: "UNPAUSED"

Niyojit

QuanDo1 · Monday

Hi balajij8, Niyojit
I understand the approach. In my opinion, this is suitable only for small number of pipelines, or local development, what if I have many pipelines, I wouldn't want them to run all at the same time, having the flexibility to customize each pipeline is I think more suitable for professional use cases?

Niyojit · Monday

What i think is for many pipelines or production-scale deployments, you would typically define pipeline-specific variables (or separate variables per workflow) and override them per target (dev, qa, prod). This gives each pipeline its own schedule while still keeping the configuration centralized.

Niyojit

balajij8 · Monday

Quan,

Instead of using global variables that apply the same schedule to all jobs, you override the schedule individually for each pipeline within each environment target. It gives you full control over every pipeline's timing in every environment.

QuanDo1 · Monday

balajij8,

Sounds promising, could you elaborate on this? Perhaps a quick example

balajij8 · Monday

Quan,

You can follow below

targets:
  prod:
    resources:
      jobs:
        job1:
          schedule:
            quartz_cron_expression: "0 0 */2 * * ?"  # Every 2 hours
        
        job2:
          schedule:
            quartz_cron_expression: "0 0 6 * * ?"    # Daily at 6am