Productionizing Databricks Pipelines with Declarative Automation Bundles and Azure DevOps

Isaac_18 — Sat, 22 Aug 2026 09:03:59 GMT

A retail medallion pipeline packaged as a bundle, promoted through Dev, QA, and Production from a single repository — including the deployment-mode bug that made every stage go green while shipping a job that could never have run.

Most Databricks projects start the same way: a notebook that works, scheduled through the workspace UI. That is fine for one pipeline. It stops being fine somewhere around the third environment and the second engineer.

We hit that wall on a retail analytics build — sales, inventory, product, and customer feeds landing into a medallion architecture, promoted across Dev, QA, and Production. This walkthrough covers how we packaged it as a Declarative Automation Bundle and deployed it from Azure DevOps, including the parts that bit us.

A note on naming

Databricks renamed Asset Bundles to Declarative Automation Bundles as part of the Lakeflow consolidation. The DAB abbreviation survives; the expansion changed.

Along with it, Workflows became Lakeflow Jobs, and Delta Live Tables became Lakeflow Spark Declarative Pipelines — often shortened to Lakeflow pipelines.

Plain "Spark Declarative Pipelines" refers to the Apache Spark open-source framework that the Databricks product extends.

Most search results and blog posts still use the older names.

1. What breaks without bundles

Three specific failures, not a general lament.

Environment drift

Dev points at retail_dev, prod at retail_prod, and the only record of that difference lives in someone's head.

A cluster configuration gets tuned in prod and never backported. Six weeks later nobody can say why QA passes and prod does not.

No change history

A UI-edited job has no diff, no author, and no reviewer.

When a pipeline breaks at 2 a.m., what changed? There is no reliable answer.

No rollback

Without a versioned definition of the job, reverting means remembering what the settings used to be.

Declarative Automation Bundles address all three by making the job definition a file in your repository.

2. What a bundle actually is

A bundle is a directory with a databricks.yml at its root that declares your Databricks resources — Lakeflow Jobs, Lakeflow pipelines, dashboards, Model Serving endpoints, MLflow experiments, and registered models — alongside the source code they run.

The Databricks CLI reads that file and reconciles your workspace to match.

You get one command per environment, and the same bundle can be promoted from Dev to Production.

A simplified repository structure looks like this:

DAB-CI-CD/ ├── resources/ │ └── retail_pipeline_job.yml ├── src/ │ └── notebooks/ │ ├── bronze/ │ │ ├── ingest_customers │ │ ├── ingest_inventory │ │ ├── ingest_products │ │ └── ingest_sales │ ├── silver/ │ │ ├── transform_customers_scd │ │ ├── transform_inventory │ │ ├── transform_products │ │ └── transform_sales │ └── gold/ │ ├── agg_store_revenue │ ├── agg_product_performance │ ├── agg_inventory_health │ └── agg_executive_summary ├── tests/ │ └── test_data_quality ├── setup_test_data.ipynb ├── azure-pipelines.yml ├── databricks.yml └── .gitignore

Nothing in that tree is created by hand in the workspace.

The job, folder layout, and notebooks are all provisioned by databricks bundle deploy.

The same structure as it appears inside the Databricks workspace after deployment. The bundle is Git-linked, so the workspace copy is a deployment target rather than a place to edit.

3. Walking through databricks.yml

Bundle identity

bundle: name: retail-modernization-dab

This name feeds the deployment path, so changing it later moves your remote state to a new location.

Pick it once.

Splitting out resources

include: - resources/*.yml

Keeping job definitions in their own files under resources/ is worth doing from day one.

A single databricks.yml holding four bronze tasks, four silver tasks, four gold tasks, and a quality check becomes unreadable fast.

Variables for what differs per environment

variables: catalog: description: "Unity Catalog name for the environment" default: "retail_dev" schema_bronze: description: "Bronze layer schema" default: "bronze" schema_silver: description: "Silver layer schema" default: "silver" schema_gold: description: "Gold layer schema" default: "gold"

The catalog is the only value that genuinely changes between environments; the layer schemas stay constant.

Declaring all four keeps the notebooks free of hardcoded names. They can read:

${var.catalog}.${var.schema_bronze}

and work anywhere.

Variable precedence

Variable precedence, highest to lowest, is worth knowing before you debug a value that "isn't taking":

--var="catalog=retail_qa" on the CLI
Environment variables prefixed BUNDLE_VAR_ — e.g. BUNDLE_VAR_catalog
A variable-overrides.json file, if present
A variables: mapping inside the target
The default: in the top-level variable definition

Where the bundle lands

workspace: root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}

This is correct for development — every engineer gets an isolated copy and nobody collides.

Do not ship this for production.

A user-scoped production path means your production pipeline is owned by one person's account and can become problematic when that person leaves the organization.

For production, deploy to a shared location and run as a service principal.

Production mode will also validate against user-scoped paths, which is covered below.

targets: # DEVELOPMENT dev: mode: development default: true workspace: host: https://YOUR-DEV-WORKSPACE.azuredatabricks.net variables: catalog: "retail_dev" # QA qa: mode: development workspace: host: https://YOUR-QA-WORKSPACE.azuredatabricks.net variables: catalog: "retail_qa" # PRODUCTION prod: mode: production workspace: host: https://YOUR-PROD-WORKSPACE.azuredatabricks.net root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target} run_as: service_principal_name: ${var.prod_service_principal} variables: catalog: "retail_prod"

The mistake we shipped

Our first working version had mode: development on all three targets.

Everything deployed cleanly and the DevOps pipeline went green, so it looked correct.

It was not.

mode: development does considerably more than its name suggests. It:

Prefixes resources with a development identifier.
Tags deployed jobs and pipelines with a development Databricks tag.
Marks Lakeflow pipelines as development.
Pauses schedules and triggers on deployed jobs and quality monitors.
Enables concurrent runs on deployed jobs.
Disables the deployment lock.
Permits cluster overrides.

So our "production" job was effectively a development deployment, with its schedule paused and its deployment lock disabled.

A green CI/CD pipeline told us nothing because deployment success is not deployment correctness.

Production mode runs the opposite set of checks.

It validates production-oriented settings such as pipeline development status, Git branch configuration, deployment paths, run-as configuration, and permissions.

It also prevents cluster overrides.

Those validations are the feature. Let them stop you.

4. The pipeline: Bronze, Silver, Gold

Four retail domains flow through three layers.

Bronze ingests with Auto Loader and stamps audit metadata.

Silver cleanses and applies merge logic.

Gold aggregates to business KPIs.

A final task validates quality before anything downstream consumes the data.

Bronze

The Bronze layer contains:

ingest_customers
ingest_inventory
ingest_products
ingest_sales

Auto Loader reads CSV files from the landing zone, adds an ingestion timestamp and source file name, and appends the data to Delta.

There is no business logic in Bronze.

Silver

The Silver layer contains:

transform_customers_scd
transform_inventory
transform_products
transform_sales

These tasks filter invalid rows, standardize identifiers to uppercase, cast types, deduplicate, and then perform MERGE operations.

Customers use SCD Type 1.

Inventory derives a reorder flag.

Gold

The Gold layer contains:

agg_store_revenue
agg_product_performance
agg_inventory_health
agg_executive_summary

These produce daily revenue and transaction counts by store, product performance by category, latest-snapshot stock health, and a company-level rollup for leadership dashboards.

Tests

The test_data_quality task checks:

Row counts
Null values
Referential integrity across the Gold tables

The Silver merges are idempotent, which matters more than it sounds.

It is what makes a re-run after a partial failure safe rather than duplicating rows.

A successful run. Bronze tasks execute in parallel; each Silver task depends only on its own Bronze task, so transform_sales starts as soon as ingest_sales finishes instead of waiting for all four. The whole run completes in under three minutes on serverless compute. Usernames and job identifiers are redacted.

Why per-task dependencies beat layer barriers

It is tempting to make all four Silver tasks depend on all four Bronze tasks.

Don't.

Wiring each Silver task to its own Bronze task means one slow domain does not hold up the other three, and a single failed ingest only blocks its own branch instead of the entire layer.

5. Azure DevOps: validate, then promote

The pipeline runs four stages on every merge to main.

Validate

Install a pinned CLI, run unit tests, then bundle validate against every target.

This catches YAML errors, missing references, and broken variable substitutions before anything is deployed.

Deploy to Dev

Run:

databricks bundle deploy --target dev

This provides fast feedback that the resources provision cleanly.

Deploy to QA

Deploy the bundle and then run the job:

databricks bundle deploy --target qa databricks bundle run retail_etl_pipeline --target qa

The data quality task has to pass here before production is reachable.

Deploy to Production

Production is gated behind an Azure DevOps environment approval check.

Promotion becomes a decision rather than a side effect of merging.

Everything the pipeline needs is in the repository — the bundle configuration, job definition, notebooks, and pipeline YAML itself.

The repository on main. Because the bundle is the deployment, every change to a job arrives as a reviewable commit. The databricks.yml and azure-pipelines.yml sit next to the notebooks they govern. Organization, project, and author details are redacted.

One structural note before the YAML: each target already declares its own workspace.host, so the pipeline passes only credentials, never DATABRICKS_HOST.

Setting a host environment variable that disagrees with the target's declared host is a common and confusing source of authentication failures.

Azure DevOps pipeline

trigger: branches: include: - main pool: vmImage: ubuntu-latest variables: DATABRICKS_CLI_VERSION: '1.13.0' stages: # CI - stage: Validate displayName: 'Validate bundle' jobs: - job: ValidateBundle steps: - checkout: self - script: | curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/v$(DATABRICKS_CLI_VERSION)/install.sh | sh databricks --version displayName: 'Install Databricks CLI' - script: | pip install -r requirements-dev.txt pytest tests/unit -q displayName: 'Unit tests' - script: | for t in dev qa prod; do echo "Validating target: $t" databricks bundle validate --target $t done displayName: 'Validate all targets' env: DATABRICKS_CLIENT_ID: $(CI_CLIENT_ID) DATABRICKS_CLIENT_SECRET: $(CI_CLIENT_SECRET) # Dev - stage: DeployDev dependsOn: Validate jobs: - deployment: DeployToDev environment: 'dev' strategy: runOnce: deploy: steps: - checkout: self - script: | curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/v$(DATABRICKS_CLI_VERSION)/install.sh | sh databricks bundle deploy --target dev displayName: 'Deploy to Dev' env: DATABRICKS_CLIENT_ID: $(DEV_CLIENT_ID) DATABRICKS_CLIENT_SECRET: $(DEV_CLIENT_SECRET) # QA - stage: DeployQA dependsOn: DeployDev jobs: - deployment: DeployToQA environment: 'qa' strategy: runOnce: deploy: steps: - checkout: self - script: | curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/v$(DATABRICKS_CLI_VERSION)/install.sh | sh databricks bundle deploy --target qa databricks bundle run retail_etl_pipeline --target qa displayName: 'Deploy and run in QA' env: DATABRICKS_CLIENT_ID: $(QA_CLIENT_ID) DATABRICKS_CLIENT_SECRET: $(QA_CLIENT_SECRET) # Production - stage: DeployProd dependsOn: DeployQA jobs: - deployment: DeployToProd environment: 'prod' strategy: runOnce: deploy: steps: - checkout: self - script: | curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/v$(DATABRICKS_CLI_VERSION)/install.sh | sh databricks bundle deploy --target prod displayName: 'Deploy to Prod' env: DATABRICKS_CLIENT_ID: $(PROD_CLIENT_ID) DATABRICKS_CLIENT_SECRET: $(PROD_CLIENT_SECRET)

One run, end to end. Commit c47cf215 on main produced four green stages in 2 minutes 36 seconds. Each stage targets a different workspace, so a failure stops promotion before it reaches the next environment. The commit hash makes any deployed state traceable back to the exact code that produced it.

Three choices worth explaining

QA runs the job, not just the deploy

Our original pipeline only ever called bundle deploy.

Every stage went green while proving nothing about whether the pipeline produced correct data.

Deploy succeeding means the configuration was valid and the resources were created.

Adding bundle run in QA means the data quality task actually has to pass before production is reachable.

Production is gated

Auto-deploying to production on every merge to main is a fast way to have a bad morning.

Azure DevOps environments support approval checks. Attach one to production and promotion becomes deliberate.

The CLI version is pinned

We originally piped install.sh from the main branch without pinning the version.

That makes CI non-reproducible.

A CLI release can change validation behavior and break a build on a commit that touched nothing.

The setup script accepts a release tag, so pin it and bump deliberately.

The result: the bundle as deployed to production, provisioned entirely by the pipeline with no manual workspace configuration. Because run_as points at a service principal, the deploy path and resource owner are the service principal rather than an individual. Path and owner are redacted here.

6. Authentication: OAuth, not PATs

Our first pass used personal access tokens per environment.

Two problems:

A PAT is tied to a human.
If it is not flagged as a secret variable in Azure DevOps, it can be exposed in the variables panel or in screenshots.

For unattended CI/CD, Databricks provides several authentication options, including managed identities and OAuth machine-to-machine authentication.

Managed identity requires an appropriate Azure-hosted execution environment. On Microsoft-hosted agents such as ubuntu-latest, OAuth M2M is a practical choice.

For OAuth M2M, generate an OAuth secret for the service principal and configure the required variables.

DATABRICKS_HOST DATABRICKS_CLIENT_ID DATABRICKS_CLIENT_SECRET

The CLI can use these through unified authentication.

Our bundle supplies the workspace host through each target, so the pipeline primarily needs to provide the credentials.

How many service principals?

Databricks supports using a service principal across participating workspaces.

Many teams instead use one service principal per environment to limit blast radius and accept the additional credential variables.

Either approach can be defensible.

Decide deliberately rather than by accident.

We use a single read-mostly service principal for validation and separate service principals for environment-specific deployments.

Three things that will bite you

Mark every credential as a secret

An Azure DevOps variable is only masked if you configure it as secret.

A plain variable holding a token can be exposed in the variables panel or captured in screenshots.

Source credentials from Azure Key Vault through a variable group where appropriate.

Unset DATABRICKS_TOKEN when migrating

A leftover DATABRICKS_TOKEN or DATABRICKS_USERNAME can conflict with OAuth and produce authentication failures that look nothing like a configuration problem.

If you are moving from PATs, remove the old variables rather than leaving them alongside the new authentication method.

OAuth secrets expire

OAuth secrets have expiration limits.

Put the expiry date in a calendar or another operational reminder.

Otherwise, your pipeline can fail on a commit that changed nothing.

7. Gotchas worth knowing first

Renaming a resource key can destroy and recreate the job

The key in resources/*.yml is the identity the CLI tracks in state.

Rename:

retail_etl_pipeline

retail_pipeline

and you can end up with a new job and a new ID.

Run history may no longer be associated with the new resource, and alerts can point at the old resource.

Change the job's display name freely; leave the resource key stable.

If you have already orphaned a resource, databricks bundle deployment bind can be used to associate an existing resource with bundle state instead of creating a duplicate.

Bundles use Terraform state

Bundle deployments maintain state.

There is remote state under the workspace root path and local bundle state associated with the target.

Deleting the workspace folder by hand, or deploying the same bundle from a machine with stale state, can result in orphaned or duplicated resources.

Add .databricks to .gitignore

The local CLI working directory should never be committed.

for example:

.databricks/ .bundle/

Deployment locks matter more than they look

Development mode disables the deployment lock.

Two engineers deploying the same development target concurrently can therefore interfere with each other.

A user-scoped development root path helps isolate individual development deployments.

Resist reaching for --force-lock after a failed pipeline run. It can create state-management problems and duplicate resources when used incorrectly.

bundle destroy is not a dry run

It removes deployed resources.

Try it in development first.

Rollback is a redeploy

Rollback is not a special command.

Check out the previous commit or tag and run:

databricks bundle deploy --target prod

again.

Tag your production releases so there is something concrete to return to.

Note that production mode validates Git-related configuration, so plan your tagging and branching strategy together.

8. Try it yourself

Install the CLI using a pinned version:

curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/v1.13.0/install.sh | sh databricks --version

Scaffold a bundle:

databricks bundle init

Authenticate as a service principal:

export DATABRICKS_HOST=https://YOUR-WORKSPACE.azuredatabricks.net export DATABRICKS_CLIENT_ID=YOUR-SERVICE-PRINCIPAL-APPLICATION-ID export DATABRICKS_CLIENT_SECRET=YOUR-OAUTH-SECRET

Validate, deploy, and run:

databricks bundle validate --target dev databricks bundle deploy --target dev databricks bundle run retail_etl_pipeline --target dev

Clean up:

databricks bundle destroy --target dev

Prerequisites

You need:

A Unity Catalog-enabled workspace
Permission to create the required catalogs and schemas
A current Databricks CLI

The CLI version used in this example is 1.13.0.

Bundle behavior and templates have changed across older CLI releases, so check:

databricks --version

before following this tutorial.

9. What we got out of it

The pipeline definition lives in Git with the code it runs.

Promotion is one command per environment.

Rollback is a checkout and redeploy.

And the mode: development bug is exactly the kind of thing a reviewer can catch in a diff and never catch in a UI.

The broader lesson from our build is simple:

A green CI/CD pipeline is not evidence that anything works.

Ours was green while deploying a production job that could never have run on schedule.

Make the pipeline assert something real — run the job, check the data — or you have automated the appearance of correctness.

Note:

Sample data in this project is synthetic. No real retail or customer data was used.

Screenshots have been redacted to remove workspace identifiers, user accounts, and credentials.

topic Re: Productionizing Databricks Pipelines with Declarative Automation Bundles and Azure DevOps in Community Articles

Productionizing Databricks Pipelines with Declarative Automation Bundles and Azure DevOps

A note on naming

1. What breaks without bundles

Environment drift

No change history

No rollback

2. What a bundle actually is

3. Walking through databricks.yml

Bundle identity

Splitting out resources

Variables for what differs per environment

Variable precedence

Where the bundle lands

The mistake we shipped

4. The pipeline: Bronze, Silver, Gold

Bronze

Silver

Gold

Tests

Why per-task dependencies beat layer barriers

5. Azure DevOps: validate, then promote

Validate

Deploy to Dev

Deploy to QA

Deploy to Production

Azure DevOps pipeline

Three choices worth explaining

QA runs the job, not just the deploy

Production is gated

The CLI version is pinned

6. Authentication: OAuth, not PATs

How many service principals?

Three things that will bite you

Mark every credential as a secret

Unset DATABRICKS_TOKEN when migrating

OAuth secrets expire

7. Gotchas worth knowing first

Renaming a resource key can destroy and recreate the job

Bundles use Terraform state

Add .databricks to .gitignore

Deployment locks matter more than they look

bundle destroy is not a dry run

Rollback is a redeploy

8. Try it yourself

Prerequisites

9. What we got out of it

Re: Productionizing Databricks Pipelines with Declarative Automation Bundles and Azure DevOps

Re: Productionizing Databricks Pipelines with Declarative Automation Bundles and Azure DevOps

Re: Productionizing Databricks Pipelines with Declarative Automation Bundles and Azure DevOps