Hi all,

Could someone clarify the intended usage of the variable-overrides.json file in Databricks Asset Bundles?

Let me give some context. Let's say my repository layout looks like this:

databricks/
├── notebooks/
│   └── notebook.ipynb
├── resources/
│   └── job.yml
├── databricks.yml

My job.yml looks somewhat like this:

  jobs:
    databricks_job:
      name: databricks_job
      max_concurrent_runs: 1

      schedule:
        quartz_cron_expression: "0 */5 * * * ?"
        timezone_id: UTC
        pause_status: ${var.pause_status}

      tasks:
        - task_key: notebook_task
          job_cluster_key: job_cluster
          notebook_task:
            notebook_path: ../notebooks/notebook.ipynb

      job_clusters:
        - job_cluster_key: job_cluster
          new_cluster:
            spark_version: 15.4.x-scala2.12
            node_type_id: ${var.node_type_id}
            data_security_mode: SINGLE_USER
            autoscale:
              min_workers: ${var.min_workers}
              max_workers: ${var.max_workers}

      parameters:
        - name: parameter_key
          default: ${var.parameter_value}

And my databricks.yml looks somewhat like this:

bundle:
  name: databricks_jobs

include:
  - resources/*.yml
  - resources/*/*.yml

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://dev-workspace.azuredatabricks.net

  prod:
    mode: production
    workspace:
      host: https://prod-workspace.azuredatabricks.net

I'm deploying via an Azure DevOps pipeline using the Databricks CLI: databricks bundle deploy --target ${{ parameters.environment }}

In reality, my setup includes multiple environments, more jobs and more parameters—such as different Storage Account names, cluster configurations, etc. I’d prefer not to overload the databricks.yml with all of these environment-specific variables.

Instead, I came across the variable-overrides.json file, which seems like a promising alternative. However, the documentation simply states: "You can also define a complex variable in the .databricks/bundle/<target>/variable-overrides.json file [...]"

Here’s where I’m stuck:

• The .databricks/ directory is excluded by .gitignore and seems to be generated only at runtime.
• Since I’m not deploying locally but via a DevOps pipeline, I’m unsure how to provide or inject these variable-overrides.json files into the .databricks/bundle/<target>/ directory during deployment.
• What’s the recommended workflow for using variable-overrides.json in a CI/CD setup like Azure DevOps?

Any insights, best practices, or examples would be much appreciated!

Thanks in advance!