How do I efficiently manage common and environment-specific job parameters in DABs

[azam-io]

I have a scenario where my Databricks asset bundles require two types of job parameters:

• Common parameters that apply to all environments
• Environment-specific parameters that differ per environment.

My current YAML setup is structured like this:

# common_parameters.yaml
common_parameters: &common_parameters
  - name: job_var
    default: "common job var value"
  - name: retry_count
    default: 3 # example common param
  - name: timeout_seconds
    default: 600 # example common param

# env_params.yaml
dev_parameters: &dev_parameters
  - name: redshift_host
    default: ${var.redshift_host}
  - name: test_param_dev # new param for testing in dev
    default: "test value dev"

prod_parameters: &prod_parameters
  - name: redshift_host
    default: ${var.redshift_host}
  - name: test_param_prod # new param for testing in prod
    default: "test value prod"

# notifications.yaml
dev_notifications: &dev_notifications
  email_notifications:
    on_failure:
      - "dev-team@example.com"

prod_notifications: &prod_notifications
  email_notifications:
    on_failure:
      - "prod-alerts@example.com"

# schedules.yaml
prod_schedule: &prod_schedule
  schedule:
    quartz_cron_expression: "0 0 0 * * ?"
    timezone_id: "UTC"
    pause_status: "PAUSED"

# main job definition and targets
resources:
  jobs:
    dummy_job:
      name: "dummy_workflow"
      edit_mode: "EDITABLE"
      tasks:
        - task_key: ingestion
          notebook_task:
            notebook_path: ../src/ingestion/test.py
            source: "WORKSPACE"

targets:
  dev:
    resources:
      jobs:
        dummy_job:
          parameters: *dev_parameters

  prod:
    resources:
      jobs:
        dummy_job:
          parameters: *prod_parameters

How do I include or merge the common parameters with the environment-specific parameters within the parameters list and is there any best practices to manage this kind of parameter reuse in Databricks asset bundle YAMLs?

[mark_ott]

To merge common parameters with environment-specific parameters in Databricks Asset Bundle (DAB) YAMLs, the most effective approach is to adopt modular YAML files, leveraging the include directive and hierarchical overrides within the targets mapping. This supports both efficient parameter reuse and scalable environment management.

Merging Parameters: Structure and Inclusion

• Define common parameters (or variables) in a separate YAML file, such as variables/common.yml, which contains settings shared across all environments.

• Environment-specific variables should each reside in dedicated files, like variables/<workflow>.dev.yml, variables/<workflow>.test.yml, variables/<workflow>.prod.yml, etc.

• In your main databricks.yml, use the include directive to bring in the common.yml and the specific environment file required for deployment. Update the included environment-specific file either manually or via automation (e.g., with a CI/CD script).

Here's some example code (text) 

include:
- resources/**/*.yml
- variables/common.yml
- variables/<workflow>.dev.yml # Switch to test.yml, prod.yml, etc. as needed