A retail medallion pipeline packaged as a bundle, promoted through Dev, QA, and Production from a single repository — including the deployment-mode bug that made every stage go green while shipping a job that could never have run.
Most Databricks projects start the same way: a notebook that works, scheduled through the workspace UI. That is fine for one pipeline. It stops being fine somewhere around the third environment and the second engineer.
We hit that wall on a retail analytics build — sales, inventory, product, and customer feeds landing into a medallion architecture, promoted across Dev, QA, and Production. This walkthrough covers how we packaged it as a Declarative Automation Bundle and deployed it from Azure DevOps, including the parts that bit us.
A note on naming
Databricks renamed Asset Bundles to Declarative Automation Bundles as part of the Lakeflow consolidation. The DAB abbreviation survives; the expansion changed.
Along with it, Workflows became Lakeflow Jobs, and Delta Live Tables became Lakeflow Spark Declarative Pipelines — often shortened to Lakeflow pipelines.
Plain "Spark Declarative Pipelines" refers to the Apache Spark open-source framework that the Databricks product extends.
Most search results and blog posts still use the older names.
1. What breaks without bundles
Three specific failures, not a general lament.
Environment drift
Dev points at retail_dev, prod at retail_prod, and the only record of that difference lives in someone's head.
A cluster configuration gets tuned in prod and never backported. Six weeks later nobody can say why QA passes and prod does not.
No change history
A UI-edited job has no diff, no author, and no reviewer.
When a pipeline breaks at 2 a.m., what changed? There is no reliable answer.
No rollback
Without a versioned definition of the job, reverting means remembering what the settings used to be.
Declarative Automation Bundles address all three by making the job definition a file in your repository.
2. What a bundle actually is
A bundle is a directory with a databricks.yml at its root that declares your Databricks resources — Lakeflow Jobs, Lakeflow pipelines, dashboards, Model Serving endpoints, MLflow experiments, and registered models — alongside the source code they run.
The Databricks CLI reads that file and reconciles your workspace to match.
You get one command per environment, and the same bundle can be promoted from Dev to Production.
A simplified repository structure looks like this:
DAB-CI-CD/
├── resources/
│ └── retail_pipeline_job.yml
├── src/
│ └── notebooks/
│ ├── bronze/
│ │ ├── ingest_customers
│ │ ├── ingest_inventory
│ │ ├── ingest_products
│ │ └── ingest_sales
│ ├── silver/
│ │ ├── transform_customers_scd
│ │ ├── transform_inventory
│ │ ├── transform_products
│ │ └── transform_sales
│ └── gold/
│ ├── agg_store_revenue
│ ├── agg_product_performance
│ ├── agg_inventory_health
│ └── agg_executive_summary
├── tests/
│ └── test_data_quality
├── setup_test_data.ipynb
├── azure-pipelines.yml
├── databricks.yml
└── .gitignoreNothing in that tree is created by hand in the workspace.
The job, folder layout, and notebooks are all provisioned by databricks bundle deploy.
The same structure as it appears inside the Databricks workspace after deployment. The bundle is Git-linked, so the workspace copy is a deployment target rather than a place to edit.
3. Walking through databricks.yml
Bundle identity
bundle:
name: retail-modernization-dabThis name feeds the deployment path, so changing it later moves your remote state to a new location.
Pick it once.
Splitting out resources
include:
- resources/*.ymlKeeping job definitions in their own files under resources/ is worth doing from day one.
A single databricks.yml holding four bronze tasks, four silver tasks, four gold tasks, and a quality check becomes unreadable fast.
Variables for what differs per environment
variables:
catalog:
description: "Unity Catalog name for the environment"
default: "retail_dev"
schema_bronze:
description: "Bronze layer schema"
default: "bronze"
schema_silver:
description: "Silver layer schema"
default: "silver"
schema_gold:
description: "Gold layer schema"
default: "gold"The catalog is the only value that genuinely changes between environments; the layer schemas stay constant.
Declaring all four keeps the notebooks free of hardcoded names. They can read:
${var.catalog}.${var.schema_bronze}and work anywhere.
Variable precedence
Variable precedence, highest to lowest, is worth knowing before you debug a value that "isn't taking":
- --var="catalog=retail_qa" on the CLI
- Environment variables prefixed BUNDLE_VAR_ — e.g. BUNDLE_VAR_catalog
- A variable-overrides.json file, if present
- A variables: mapping inside the target
- The default: in the top-level variable definition
Where the bundle lands
workspace:
root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}This is correct for development — every engineer gets an isolated copy and nobody collides.
Do not ship this for production.
A user-scoped production path means your production pipeline is owned by one person's account and can become problematic when that person leaves the organization.
For production, deploy to a shared location and run as a service principal.
Production mode will also validate against user-scoped paths, which is covered below.
targets:
# DEVELOPMENT
dev:
mode: development
default: true
workspace:
host: https://YOUR-DEV-WORKSPACE.azuredatabricks.net
variables:
catalog: "retail_dev"
# QA
qa:
mode: development
workspace:
host: https://YOUR-QA-WORKSPACE.azuredatabricks.net
variables:
catalog: "retail_qa"
# PRODUCTION
prod:
mode: production
workspace:
host: https://YOUR-PROD-WORKSPACE.azuredatabricks.net
root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target}
run_as:
service_principal_name: ${var.prod_service_principal}
variables:
catalog: "retail_prod"The mistake we shipped
Our first working version had mode: development on all three targets.
Everything deployed cleanly and the DevOps pipeline went green, so it looked correct.
It was not.
mode: development does considerably more than its name suggests. It:
- Prefixes resources with a development identifier.
- Tags deployed jobs and pipelines with a development Databricks tag.
- Marks Lakeflow pipelines as development.
- Pauses schedules and triggers on deployed jobs and quality monitors.
- Enables concurrent runs on deployed jobs.
- Disables the deployment lock.
- Permits cluster overrides.
So our "production" job was effectively a development deployment, with its schedule paused and its deployment lock disabled.
A green CI/CD pipeline told us nothing because deployment success is not deployment correctness.
Production mode runs the opposite set of checks.
It validates production-oriented settings such as pipeline development status, Git branch configuration, deployment paths, run-as configuration, and permissions.
It also prevents cluster overrides.
Those validations are the feature. Let them stop you.
4. The pipeline: Bronze, Silver, Gold
Four retail domains flow through three layers.
Bronze ingests with Auto Loader and stamps audit metadata.
Silver cleanses and applies merge logic.
Gold aggregates to business KPIs.
A final task validates quality before anything downstream consumes the data.
Bronze
The Bronze layer contains:
- ingest_customers
- ingest_inventory
- ingest_products
- ingest_sales
Auto Loader reads CSV files from the landing zone, adds an ingestion timestamp and source file name, and appends the data to Delta.
There is no business logic in Bronze.
Silver
The Silver layer contains:
- transform_customers_scd
- transform_inventory
- transform_products
- transform_sales
These tasks filter invalid rows, standardize identifiers to uppercase, cast types, deduplicate, and then perform MERGE operations.
Customers use SCD Type 1.
Inventory derives a reorder flag.
Gold
The Gold layer contains:
- agg_store_revenue
- agg_product_performance
- agg_inventory_health
- agg_executive_summary
These produce daily revenue and transaction counts by store, product performance by category, latest-snapshot stock health, and a company-level rollup for leadership dashboards.
Tests
The test_data_quality task checks:
- Row counts
- Null values
- Referential integrity across the Gold tables
The Silver merges are idempotent, which matters more than it sounds.
It is what makes a re-run after a partial failure safe rather than duplicating rows.
A successful run. Bronze tasks execute in parallel; each Silver task depends only on its own Bronze task, so transform_sales starts as soon as ingest_sales finishes instead of waiting for all four. The whole run completes in under three minutes on serverless compute. Usernames and job identifiers are redacted.
Why per-task dependencies beat layer barriers
It is tempting to make all four Silver tasks depend on all four Bronze tasks.
Don't.
Wiring each Silver task to its own Bronze task means one slow domain does not hold up the other three, and a single failed ingest only blocks its own branch instead of the entire layer.
5. Azure DevOps: validate, then promote
The pipeline runs four stages on every merge to main.
Validate
Install a pinned CLI, run unit tests, then bundle validate against every target.
This catches YAML errors, missing references, and broken variable substitutions before anything is deployed.
Deploy to Dev
Run:
databricks bundle deploy --target devThis provides fast feedback that the resources provision cleanly.
Deploy to QA
Deploy the bundle and then run the job:
databricks bundle deploy --target qa
databricks bundle run retail_etl_pipeline --target qaThe data quality task has to pass here before production is reachable.
Deploy to Production
Production is gated behind an Azure DevOps environment approval check.
Promotion becomes a decision rather than a side effect of merging.
Everything the pipeline needs is in the repository — the bundle configuration, job definition, notebooks, and pipeline YAML itself.
The repository on main. Because the bundle is the deployment, every change to a job arrives as a reviewable commit. The databricks.yml and azure-pipelines.yml sit next to the notebooks they govern. Organization, project, and author details are redacted.
One structural note before the YAML: each target already declares its own workspace.host, so the pipeline passes only credentials, never DATABRICKS_HOST.
Setting a host environment variable that disagrees with the target's declared host is a common and confusing source of authentication failures.
Azure DevOps pipeline
trigger:
branches:
include:
- main
pool:
vmImage: ubuntu-latest
variables:
DATABRICKS_CLI_VERSION: '1.13.0'
stages:
# CI
- stage: Validate
displayName: 'Validate bundle'
jobs:
- job: ValidateBundle
steps:
- checkout: self
- script: |
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/v$(DATABRICKS_CLI_VERSION)/install.sh | sh
databricks --version
displayName: 'Install Databricks CLI'
- script: |
pip install -r requirements-dev.txt
pytest tests/unit -q
displayName: 'Unit tests'
- script: |
for t in dev qa prod; do
echo "Validating target: $t"
databricks bundle validate --target $t
done
displayName: 'Validate all targets'
env:
DATABRICKS_CLIENT_ID: $(CI_CLIENT_ID)
DATABRICKS_CLIENT_SECRET: $(CI_CLIENT_SECRET)
# Dev
- stage: DeployDev
dependsOn: Validate
jobs:
- deployment: DeployToDev
environment: 'dev'
strategy:
runOnce:
deploy:
steps:
- checkout: self
- script: |
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/v$(DATABRICKS_CLI_VERSION)/install.sh | sh
databricks bundle deploy --target dev
displayName: 'Deploy to Dev'
env:
DATABRICKS_CLIENT_ID: $(DEV_CLIENT_ID)
DATABRICKS_CLIENT_SECRET: $(DEV_CLIENT_SECRET)
# QA
- stage: DeployQA
dependsOn: DeployDev
jobs:
- deployment: DeployToQA
environment: 'qa'
strategy:
runOnce:
deploy:
steps:
- checkout: self
- script: |
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/v$(DATABRICKS_CLI_VERSION)/install.sh | sh
databricks bundle deploy --target qa
databricks bundle run retail_etl_pipeline --target qa
displayName: 'Deploy and run in QA'
env:
DATABRICKS_CLIENT_ID: $(QA_CLIENT_ID)
DATABRICKS_CLIENT_SECRET: $(QA_CLIENT_SECRET)
# Production
- stage: DeployProd
dependsOn: DeployQA
jobs:
- deployment: DeployToProd
environment: 'prod'
strategy:
runOnce:
deploy:
steps:
- checkout: self
- script: |
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/v$(DATABRICKS_CLI_VERSION)/install.sh | sh
databricks bundle deploy --target prod
displayName: 'Deploy to Prod'
env:
DATABRICKS_CLIENT_ID: $(PROD_CLIENT_ID)
DATABRICKS_CLIENT_SECRET: $(PROD_CLIENT_SECRET)
One run, end to end. Commit c47cf215 on main produced four green stages in 2 minutes 36 seconds. Each stage targets a different workspace, so a failure stops promotion before it reaches the next environment. The commit hash makes any deployed state traceable back to the exact code that produced it.
Three choices worth explaining
QA runs the job, not just the deploy
Our original pipeline only ever called bundle deploy.
Every stage went green while proving nothing about whether the pipeline produced correct data.
Deploy succeeding means the configuration was valid and the resources were created.
Adding bundle run in QA means the data quality task actually has to pass before production is reachable.
Production is gated
Auto-deploying to production on every merge to main is a fast way to have a bad morning.
Azure DevOps environments support approval checks. Attach one to production and promotion becomes deliberate.
The CLI version is pinned
We originally piped install.sh from the main branch without pinning the version.
That makes CI non-reproducible.
A CLI release can change validation behavior and break a build on a commit that touched nothing.
The setup script accepts a release tag, so pin it and bump deliberately.
The result: the bundle as deployed to production, provisioned entirely by the pipeline with no manual workspace configuration. Because run_as points at a service principal, the deploy path and resource owner are the service principal rather than an individual. Path and owner are redacted here.
6. Authentication: OAuth, not PATs
Our first pass used personal access tokens per environment.
Two problems:
- A PAT is tied to a human.
- If it is not flagged as a secret variable in Azure DevOps, it can be exposed in the variables panel or in screenshots.
For unattended CI/CD, Databricks provides several authentication options, including managed identities and OAuth machine-to-machine authentication.
Managed identity requires an appropriate Azure-hosted execution environment. On Microsoft-hosted agents such as ubuntu-latest, OAuth M2M is a practical choice.
For OAuth M2M, generate an OAuth secret for the service principal and configure the required variables.
DATABRICKS_HOST
DATABRICKS_CLIENT_ID
DATABRICKS_CLIENT_SECRETThe CLI can use these through unified authentication.
Our bundle supplies the workspace host through each target, so the pipeline primarily needs to provide the credentials.
How many service principals?
Databricks supports using a service principal across participating workspaces.
Many teams instead use one service principal per environment to limit blast radius and accept the additional credential variables.
Either approach can be defensible.
Decide deliberately rather than by accident.
We use a single read-mostly service principal for validation and separate service principals for environment-specific deployments.
Three things that will bite you
Mark every credential as a secret
An Azure DevOps variable is only masked if you configure it as secret.
A plain variable holding a token can be exposed in the variables panel or captured in screenshots.
Source credentials from Azure Key Vault through a variable group where appropriate.
Unset DATABRICKS_TOKEN when migrating
A leftover DATABRICKS_TOKEN or DATABRICKS_USERNAME can conflict with OAuth and produce authentication failures that look nothing like a configuration problem.
If you are moving from PATs, remove the old variables rather than leaving them alongside the new authentication method.
OAuth secrets expire
OAuth secrets have expiration limits.
Put the expiry date in a calendar or another operational reminder.
Otherwise, your pipeline can fail on a commit that changed nothing.
7. Gotchas worth knowing first
Renaming a resource key can destroy and recreate the job
The key in resources/*.yml is the identity the CLI tracks in state.
Rename:
retail_etl_pipelineto
retail_pipelineand you can end up with a new job and a new ID.
Run history may no longer be associated with the new resource, and alerts can point at the old resource.
Change the job's display name freely; leave the resource key stable.
If you have already orphaned a resource, databricks bundle deployment bind can be used to associate an existing resource with bundle state instead of creating a duplicate.
Bundles use Terraform state
Bundle deployments maintain state.
There is remote state under the workspace root path and local bundle state associated with the target.
Deleting the workspace folder by hand, or deploying the same bundle from a machine with stale state, can result in orphaned or duplicated resources.
Add .databricks to .gitignore
The local CLI working directory should never be committed.
for example:
.databricks/
.bundle/Deployment locks matter more than they look
Development mode disables the deployment lock.
Two engineers deploying the same development target concurrently can therefore interfere with each other.
A user-scoped development root path helps isolate individual development deployments.
Resist reaching for --force-lock after a failed pipeline run. It can create state-management problems and duplicate resources when used incorrectly.
bundle destroy is not a dry run
It removes deployed resources.
Try it in development first.
Rollback is a redeploy
Rollback is not a special command.
Check out the previous commit or tag and run:
databricks bundle deploy --target prodagain.
Tag your production releases so there is something concrete to return to.
Note that production mode validates Git-related configuration, so plan your tagging and branching strategy together.
8. Try it yourself
Install the CLI using a pinned version:
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/v1.13.0/install.sh | sh
databricks --versionScaffold a bundle:
databricks bundle initAuthenticate as a service principal:
export DATABRICKS_HOST=https://YOUR-WORKSPACE.azuredatabricks.net
export DATABRICKS_CLIENT_ID=YOUR-SERVICE-PRINCIPAL-APPLICATION-ID
export DATABRICKS_CLIENT_SECRET=YOUR-OAUTH-SECRETValidate, deploy, and run:
databricks bundle validate --target dev
databricks bundle deploy --target dev
databricks bundle run retail_etl_pipeline --target devClean up:
databricks bundle destroy --target devPrerequisites
You need:
- A Unity Catalog-enabled workspace
- Permission to create the required catalogs and schemas
- A current Databricks CLI
The CLI version used in this example is 1.13.0.
Bundle behavior and templates have changed across older CLI releases, so check:
databricks --versionbefore following this tutorial.
9. What we got out of it
The pipeline definition lives in Git with the code it runs.
Promotion is one command per environment.
Rollback is a checkout and redeploy.
And the mode: development bug is exactly the kind of thing a reviewer can catch in a diff and never catch in a UI.
The broader lesson from our build is simple:
A green CI/CD pipeline is not evidence that anything works.
Ours was green while deploying a production job that could never have run on schedule.
Make the pipeline assert something real — run the job, check the data — or you have automated the appearance of correctness.
Note:
Sample data in this project is synthetic. No real retail or customer data was used.
Screenshots have been redacted to remove workspace identifiers, user accounts, and credentials.
