Databricks Community

SrinuB · Monday

DevOps · CICD · Deploy the same definition to dev, staging and prod, and never edit a job in the UI again

By @SrinuB in collaboration with @sridharplv

Somebody "just fixed" a production job in the UI on Friday evening. By Monday it no longer matches the repo, nobody remembers what changed, and the dev copy behaves differently from prod.

This is configuration drift, and it is the problem CICD exists to remove. On Databricks the recommended way to do it is Declarative Automation Bundles (formerly Databricks Asset Bundles, DABs). The docs state that bundles are the recommended approach to CICD on Databricks.

What a bundle is

A bundle describes your jobs, Lakeflow pipelines and other resources as source files, next to your code, in Git. The CLI validates the definition and deploys it to a workspace. The same files go to every environment. Only the target changes.

One file, three targets

bundle:
  name: customer_pipeline

include:
  - resources/*.yml

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: <dev-workspace-url>

  staging:
    workspace:
      host: <staging-workspace-url>

  prod:
    mode: production
    workspace:
      host: <prod-workspace-url>
    run_as:
      service_principal_name: <service-principal-application-id>

The jobs themselves live in resources/ and are shared by all three targets. Per-environment values (catalog names, cluster sizes, schedules) are overrides on the target, not copies of the job.

What the two modes do for you

From the docs, development mode:

Prefixes deployed resources with [dev <your short name>], so developers do not collide
Tags jobs and pipelines with dev
Pauses all schedules and triggers
Enables concurrent runs on jobs

Production mode adds guard rails instead:

Deployment fails validation unless Lakeflow pipelines are development: false
If the target sets git.branch, the current branch must match it
Cluster overrides on deploy are not allowed
run_as should be a service principal, which the docs recommend

The pipeline, step by step

Pull request: lint, run unit tests with pytest, then databricks bundle validate. Block the merge if anything fails.
Merge to main: databricks bundle deploy -t staging, then databricks bundle run -t staging <test_job>. A failed smoke test stops the release.
Approval, then prod: a manual gate (GitHub environment protection or an Azure DevOps approval), then databricks bundle deploy -t prod.
Monitor: alerts on job failure. To roll back, redeploy the previous Git tag.

The docs list Azure DevOps, GitHub Actions and Jenkins as the usual orchestrators. The bundle does not care which one you pick.

Authentication: no personal tokens

The docs say to use service principals, not users, for CICD. For the credential, the recommended option is workload identity federation, which removes the need to store a Databricks secret at all. In GitHub Actions that means id-token: write permissions plus these environment variables:

permissions:
  id-token: write
  contents: read

steps:
  - uses: actions/checkout@v4
  - uses: databricks/setup-cli@main
  - run: databricks bundle deploy -t prod
    env:
      DATABRICKS_AUTH_TYPE: github-oidc
      DATABRICKS_HOST: <prod-workspace-url>
      DATABRICKS_CLIENT_ID: <service-principal-application-id>

You create a federation policy on the service principal whose subject matches the GitHub token subject. A client-secret token stored as a GitHub secret also works, but it is the second choice.

What to watch out for

Bundle identity is name, target and deployer. Two bundles sharing all three overwrite each other's deployments.
bundle destroy is permanent. It deletes the deployed jobs, pipelines and artifacts. Use --auto-approve only in throwaway dev targets.
Prod stays read-only. The docs recommend deploying to a read-only folder so non-admins cannot change deployed assets.
Unit tests are not enough. Opinion: the staging run on realistic data is the step that catches real failures. Keep it.

A 20-minute check you can run this week

Pick one job that was created in the UI
Run databricks bundle init and recreate it as a resource file
Deploy it to a dev target and compare the run with the original
Add databricks bundle validate to a pull request check

The takeaway

Everything is code. Jobs and pipelines live in Git and go through review.
One definition, many targets. Environments differ by overrides, not by copies.
Use a service principal, ideally with workload identity federation.
Gate production. Validate in the pull request, test in staging, approve, then deploy.

Which job in your workspace would you move into a bundle first?