topic Re: DAB best practices suggestion in Data Engineering

DAB best practices suggestion

DazzaiDe — Sat, 27 Jun 2026 16:13:18 GMT

We're currently setting up Databricks Asset Bundles (DAB) with a CI/CD pipeline using Azure DevOps.

Our planned development workflow is as follows:

Main branch → Developer creates a feature branch → Implement changes → Create a Pull Request → Senior developers review and approve → Merge into the main branch → Deploy to UAT → After UAT sign-off, deploy to Production.

I would like to hear suggestions specially the best practices as of now

Re: DAB best practices suggestion

balajij8 — Sat, 27 Jun 2026 17:07:22 GMT

You can create Databricks Asset Bundles that are decoupled by domain, managed via multi target declarations within configuration and also driven by immutable, versioned artifacts stored securely within Unity Catalog Volumes. You can rely on explicit CI/CD gating and dynamic, scoped resource names rather than monolithic & hardcoded infrastructure definitions.

Bundle Structure & Domain Isolation

Decoupled Domain Bundles - You can group configurations into small focused bundles aligned to specific data products or business domains instead of monolithic setup.
Shared Lifecycles - Ensure that a single bundle contains only the resources (jobs, pipelines, dashboards) that share a unified deployment lifecycle and ownership domain boundary.
Target Definitions - You can maintain all target definitions (dev, uat, prod) within a single yml per bundle to guarantee environmental structural parity. More details here

Multi-Target Environment Strategy

Development - Configure for feature-branch agility. Implement dynamic resource renaming using built-in metadata expressions (such as ${workspace.current_user.short_name}) to enforce isolation within shared or personal workspaces. Route all computation to development catalogs and schemas.
Staging/User Acceptance Testing - Trigger automated deployments on pull request merges to the main branch. This layer must run full integration suites and validation workflows against pre-production catalogs, mirroring production configurations identically.
Production - Guard production workloads with manual approval workflows and strict role-based access control (RBAC) with the target production Unity Catalog environments.

CI/CD Orchestration (Azure DevOps)

Pull Request Verification - Enforce static analysis by running databricks bundle validate prior to any code merges to catch syntactical and structural anomalies early.
Continuous Deployment (UAT) - Compile code, version artifacts, stage them directly into Unity Catalog volumes and execute target-specific deployments sequentially.
Release Management (Prod) - Restrict production deployments to manual approval gates within Azure DevOps Environments. Re-use the identical, immutable artifacts verified in UAT to eliminate drift.

Artifact & Dependency Management

Unity Catalog Volumes - Store external dependencies (Python Wheels, JARs) inside secure, governed Unity Catalog Volumes rather than embedding large binaries directly into the bundle workspace.
Inter-Bundle Governance - Model complex cross-bundle dependencies explicitly within Azure DevOps YAML pipeline tasks rather than nesting configuration files. Fail pipeline execution immediately if upstream assets are absent.

Re: DAB best practices suggestion

savlahanish27 — Tue, 30 Jun 2026 10:11:16 GMT

The workflow itself is solid. A few things worth tightening at each stage, based on running this in production with Azure DevOps:

At the PR stage, run databricks bundle validate as a required check before merge is allowed. It catches wrong field names, broken resource references, and missing values before anything touches a workspace. Costs nothing to run and it's the cheapest gate in the whole pipeline.

For your dev target, make sure databricks.yml has mode: development with default: true. This prefixes resource names with the username automatically, so a few people can deploy to the same dev workspace without stepping on each other. Skip default: true and someone in a hurry will eventually run bundle deploy with no --target flag and end up somewhere they didn't mean to.

For UAT and prod, use mode: production and deploy under a service principal, not a personal account. Two reasons this matters: deployed resources don't get orphaned if someone leaves the team, and your prod audit trail actually means something when someone asks who deployed what.

Worth testing deliberately before you need it - what happens if a deploy fails mid-run and you have to retry. There's a thread on this board right now about --force-lock creating duplicate jobs after an Azure DevOps pipeline failure. Simulate that failure in dev before it happens for real in prod.

And keep validate as a gate at every promotion, not just at the PR. The bigger thing though is to promote the same validated artifact from UAT into prod rather than rebuilding from YAML at each stage - that's what actually guarantees prod matches what UAT tested.

Putting it together:

Feature branch → validate (PR gate) → review and merge → validate again → deploy to UAT under service principal → UAT sign-off → promote the same artifact to prod, don't rebuild from source → deploy to prod under service principal with manual approval

The part that matters most: validate early and often, but only ever promote one artifact forward instead of re-deploying from source at each stage. That's what keeps UAT and prod actually identical to what got tested.

Happy to go deeper on any of this if it's useful for what you're setting up.