cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results forย 
Search instead forย 
Did you mean:ย 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results forย 
Search instead forย 
Did you mean:ย 

DAB best practices suggestion

DazzaiDe
New Contributor III

Saturday

We're currently setting up Databricks Asset Bundles (DAB) with a CI/CD pipeline using Azure DevOps.

Our planned development workflow is as follows:

Main branch โ†’ Developer creates a feature branch โ†’ Implement changes โ†’ Create a Pull Request โ†’ Senior developers review and approve โ†’ Merge into the main branch โ†’ Deploy to UAT โ†’ After UAT sign-off, deploy to Production.

I would like to hear suggestions specially the best practices as of now

Labels:
0 Kudos
1 REPLY 1

balajij8
Contributor III

Saturday

You can create Databricks Asset Bundles that are decoupled by domain, managed via multi target declarations within configuration and also driven by immutable, versioned artifacts stored securely within Unity Catalog Volumes. You can rely on explicit CI/CD gating and dynamic, scoped resource names rather than monolithic & hardcoded infrastructure definitions.

Bundle Structure & Domain Isolation

  • Decoupled Domain Bundles - You can group configurations into small focused bundles aligned to specific data products or business domains instead of monolithic setup.

  • Shared Lifecycles - Ensure that a single bundle contains only the resources (jobs, pipelines, dashboards) that share a unified deployment lifecycle and ownership domain boundary.

  • Target Definitions - You can maintain all target definitions (dev, uat, prod) within a single yml per bundle to guarantee environmental structural parity. More details here

Multi-Target Environment Strategy

  • Development - Configure for feature-branch agility. Implement dynamic resource renaming using built-in metadata expressions (such as ${workspace.current_user.short_name}) to enforce isolation within shared or personal workspaces. Route all computation to development catalogs and schemas.

  • Staging/User Acceptance Testing - Trigger automated deployments on pull request merges to the main branch. This layer must run full integration suites and validation workflows against pre-production catalogs, mirroring production configurations identically.

  • Production - Guard production workloads with manual approval workflows and strict role-based access control (RBAC) with the target production Unity Catalog environments.

CI/CD Orchestration (Azure DevOps)

  • Pull Request Verification - Enforce static analysis by running databricks bundle validate prior to any code merges to catch syntactical and structural anomalies early.

  • Continuous Deployment (UAT) - Compile code, version artifacts, stage them directly into Unity Catalog volumes and execute target-specific deployments sequentially.

  • Release Management (Prod) - Restrict production deployments to manual approval gates within Azure DevOps Environments. Re-use the identical, immutable artifacts verified in UAT to eliminate drift.

Artifact & Dependency Management

  • Unity Catalog Volumes - Store external dependencies (Python Wheels, JARs) inside secure, governed Unity Catalog Volumes rather than embedding large binaries directly into the bundle workspace.

  • Inter-Bundle Governance - Model complex cross-bundle dependencies explicitly within Azure DevOps YAML pipeline tasks rather than nesting configuration files. Fail pipeline execution immediately if upstream assets are absent.

0 Kudos
Announcements