You can configure the runtime SP to create tables dynamically on the first run if you prioritize operational simplicity and self-contained pipelines. The data processing scripts inject CREATE TABLE IF NOT EXISTS statements just before executing Data Manipulation Language (DML) operations in this approach. On the initial run, the runtime SP provisions the tables & on all subsequent runs, the DDL is safely bypassed. This approach keeps the schema definitions version-controlled alongside your application logic, eliminating complex orchestration. The runtime SP will require explicit CREATE TABLE and USE SCHEMA permissions on the target Unity Catalog schema.
You can deploy a dedicated initialization job if the governance environment demands strict separation of concerns or operates under rigorous compliance constraints. In this approach, the bundle defines two separate workflows - a setup job explicitly tasked with executing idempotent DDL statements and the primary data processing jobs. By orchestrating the initialization job to run first (potentially under a more privileged SP), you can restrict the primary runtime SP to only SELECT and MODIFY privileges. This strictly enforces the best practices while separating infrastructure provisioning from data transformation.
You can create Bundles - DAB that are decoupled by domain, managed via multi target declarations within configuration and also driven by immutable, versioned artifacts stored securely within Unity Catalog Volumes. You can rely on explicit CI/CD gating and dynamic, scoped resource names rather than monolithic & hardcoded infrastructure definitions.
Bundle Structure & Domain Isolation
Decoupled Domain Bundles - You can group configurations into small focused bundles aligned to specific data products or business domains instead of monolithic setup.
Shared Lifecycles - Ensure that a single bundle contains only the resources (jobs, pipelines, dashboards) that share a unified deployment lifecycle and ownership domain boundary. More details here
Target Definitions - You can maintain all target definitions (dev, uat, prod) within a single yml per bundle to guarantee environmental structural parity. More details here
Multi-Target Environment Strategy
Development - Configure for feature-branch agility. Implement dynamic resource renaming using built-in metadata expressions to enforce isolation within shared or personal workspaces. Route all computation to development catalogs and schemas.
Staging/User Acceptance Testing - Trigger automated deployments on pull request merges to the main branch. This layer must run full integration suites and validation workflows against pre-production catalogs, mirroring production configurations identically.
Production - Guard production workloads with manual approval workflows and strict role-based access control (RBAC) with the target production Unity Catalog environments.
CI/CD Orchestration (Azure DevOps)
Pull Request Verification - Enforce static analysis by running databricks bundle validate prior to any code merges to catch syntactical and structural anomalies early.
Continuous Deployment (UAT) - Compile code, version artifacts, stage them directly into Unity Catalog volumes and execute target-specific deployments sequentially.
Release Management (Prod) - Restrict production deployments to manual approval gates within Azure DevOps Environments. Re-use the identical, immutable artifacts verified in UAT to eliminate drift.
- Dev to Stage - Code Push to main branch can trigger deployment to Stage/QA
- Stage to Prod - After successful Stage deployment, you can trigger Prod deployment after manual approval.
Artifact & Dependency Management
Unity Catalog Volumes - Store external dependencies (Python Wheels, JARs) inside secure, governed Unity Catalog Volumes rather than embedding large binaries directly into the bundle workspace.
Inter-Bundle Governance - Model complex cross-bundle dependencies explicitly within Azure DevOps YAML pipeline tasks rather than nesting configuration files. Fail pipeline execution immediately if upstream assets are absent.
The first-run dynamic creation method is typically the optimal choice as it minimizes moving parts, guarantees that your schemas and code are perfectly synchronized and avoids the overhead of managing secondary setup jobs.
