Asset Bundles Setup

[Cristian17]

Hi everyone,

I have a question about the best practice for organizing Databricks Asset Bundles.

I have multiple projects, and each project has its own notebooks and jobs.

Is it recommended to:

create a single bundle that includes all projects, or
create a separate bundle for each project?

Also, my understanding is that the Databricks UI doesn't support deploying only selected resources from a bundle, while the Databricks CLI allows this (for example, using bundle deploy --select).

Does this limitation influence how you structure your bundles? What approach has worked best for your team?

[balajij8]

You can create Bundles that are decoupled by domain, managed via multi target declarations within configuration and also driven by immutable, versioned artifacts stored securely within Unity Catalog Volumes. You can rely on explicit CI/CD gating and dynamic, scoped resource names rather than monolithic & hardcoded infrastructure definitions.

Bundle Structure & Domain Isolation

• Decoupled Domain Bundles - You can group configurations into small focused bundles aligned to specific data products or business domains instead of monolithic setup.

• Shared Lifecycles - Ensure that a single bundle contains only the resources (jobs, pipelines, dashboards) that share a unified deployment lifecycle and ownership domain boundary.

• Target Definitions - You can maintain all target definitions (dev, uat, prod) within a single yml per bundle to guarantee environmental structural parity. More details here

Multi-Target Environment Strategy

• Development - Configure for feature-branch agility. Implement dynamic resource renaming using built-in metadata expressions (such as ${workspace.current_user.short_name}) to enforce isolation within shared or personal workspaces. Route all computation to development catalogs and schemas.

• Staging/User Acceptance Testing - Trigger automated deployments on pull request merges to the main branch. This layer must run full integration suites and validation workflows against pre-production catalogs, mirroring production configurations identically.

• Production - Guard production workloads with manual approval workflows and strict role-based access control (RBAC) with the target production Unity Catalog environments.

CI/CD Orchestration (Azure DevOps)

• Pull Request Verification - Enforce static analysis by running databricks bundle validate prior to any code merges to catch syntactical and structural anomalies early.

• Continuous Deployment (UAT) - Compile code, version artifacts, stage them directly into Unity Catalog volumes and execute target-specific deployments sequentially.

• Release Management (Prod) - Restrict production deployments to manual approval gates within Azure DevOps Environments. Re-use the identical, immutable artifacts verified in UAT to eliminate drift.

Artifact & Dependency Management

• Unity Catalog Volumes - Store external dependencies (Python Wheels, JARs) inside secure, governed Unity Catalog Volumes rather than embedding large binaries directly into the bundle workspace.

• Inter-Bundle Governance - Model complex cross-bundle dependencies explicitly within Azure DevOps YAML pipeline tasks rather than nesting configuration files. Fail pipeline execution immediately if upstream assets are absent.