How do I efficiently manage common and environment-specific job parameters in DABs

azam-io — Fri, 01 Aug 2025 13:09:40 GMT

I have a scenario where my Databricks asset bundles require two types of job parameters:

Common parameters that apply to all environments
Environment-specific parameters that differ per environment.

My current YAML setup is structured like this:

# common_parameters.yaml common_parameters: &common_parameters - name: job_var default: "common job var value" - name: retry_count default: 3 # example common param - name: timeout_seconds default: 600 # example common param # env_params.yaml dev_parameters: &dev_parameters - name: redshift_host default: ${var.redshift_host} - name: test_param_dev # new param for testing in dev default: "test value dev" prod_parameters: &prod_parameters - name: redshift_host default: ${var.redshift_host} - name: test_param_prod # new param for testing in prod default: "test value prod" # notifications.yaml dev_notifications: &dev_notifications email_notifications: on_failure: - "dev-team@example.com" prod_notifications: &prod_notifications email_notifications: on_failure: - "prod-alerts@example.com" # schedules.yaml prod_schedule: &prod_schedule schedule: quartz_cron_expression: "0 0 0 * * ?" timezone_id: "UTC" pause_status: "PAUSED" # main job definition and targets resources: jobs: dummy_job: name: "dummy_workflow" edit_mode: "EDITABLE" tasks: - task_key: ingestion notebook_task: notebook_path: ../src/ingestion/test.py source: "WORKSPACE" targets: dev: resources: jobs: dummy_job: parameters: *dev_parameters prod: resources: jobs: dummy_job: parameters: *prod_parameters

How do I include or merge the common parameters with the environment-specific parameters within the parameters list and is there any best practices to manage this kind of parameter reuse in Databricks asset bundle YAMLs?

Re: How do I efficiently manage common and environment-specific job parameters in DABs

mark_ott — Mon, 22 Sep 2025 18:50:37 GMT

To merge common parameters with environment-specific parameters in Databricks Asset Bundle (DAB) YAMLs, the most effective approach is to adopt modular YAML files, leveraging the include directive and hierarchical overrides within the targets mapping. This supports both efficient parameter reuse and scalable environment management.

Merging Parameters: Structure and Inclusion

Define common parameters (or variables) in a separate YAML file, such as variables/common.yml, which contains settings shared across all environments.
Environment-specific variables should each reside in dedicated files, like variables/<workflow>.dev.yml, variables/<workflow>.test.yml, variables/<workflow>.prod.yml, etc.
In your main databricks.yml, use the include directive to bring in the common.yml and the specific environment file required for deployment. Update the included environment-specific file either manually or via automation (e.g., with a CI/CD script).

Here's some example code (text)

include:
- resources/**/*.yml
- variables/common.yml
- variables/<workflow>.dev.yml # Switch to test.yml, prod.yml, etc. as needed

topic How do I efficiently manage common and environment-specific job parameters in DABs in Data Engineering

How do I efficiently manage common and environment-specific job parameters in DABs

Re: How do I efficiently manage common and environment-specific job parameters in DABs

Merging Parameters: Structure and Inclusion