Monorepo vs Multi-repo for Databricks Asset Bundles: A Decision Framework

[DoTA]

When teams scale their Databricks Asset Bundles (DABs) adoption, one of the first infrastructure questions that comes up is: should we keep all our bundles in a single monorepo, or split them across dedicated repos per domain? Having run both models at a regulated financial institution with roughly 30 engineers across 5 data domains, here is the decision framework we landed on.

The Core Trade-off

Monorepo gives you a single source of truth for shared libraries, notebooks, and configuration variables. Multi-repo gives you clean deployment boundaries, independent release cycles, and fine-grained access control per domain. Neither is universally better — the right answer depends on four factors.

Decision Factor 1: Team Size and Domain Count

• Fewer than 20 engineers, fewer than 4 domains: Monorepo. The overhead of cross-repo dependency management costs more than you gain.
• 20–50 engineers, 4–8 domains: Monorepo with clear bundle boundaries. This is the sweet spot where shared libraries deliver the most leverage.
• More than 50 engineers or more than 8 domains: Consider splitting. Coordination cost grows super-linearly at this scale.

Decision Factor 2: Release Cadence

• All domains ship on a unified release cycle: Monorepo. Synchronized releases are simpler to manage from one place.
• Domains have independent release cycles (e.g. Risk deploys weekly, Marketing deploys daily): Multi-repo. Coupling release pipelines across domains creates unnecessary blocking.

Decision Factor 3: Access Control Requirements

• All engineers can see all code: Monorepo. No complications here.
• Some domains contain sensitive models or regulated data logic visible only to specific teams: Multi-repo. Repo-level access control is simpler than implementing fine-grained path-level controls within a monorepo.

Decision Factor 4: CI Pipeline Speed

• Full CI runs in under 15 minutes: Monorepo is fine. Every push validates everything together.
• CI takes longer than 15 minutes: Path-based filtering becomes mandatory. If your CI tooling does not support it well, consider splitting.

Decision Summary

Start with monorepo if: team is under 50 engineers, domains share a release cycle, no strict access control boundaries exist, and CI runs in under 15 minutes. Move toward multi-repo when: teams exceed 50 engineers, domains need independent deployments, access control becomes a compliance requirement, or CI times exceed 15 minutes despite path filtering.

Making Monorepo Work: Three Shared-Asset Patterns

If you go monorepo, the three main shared-asset problems are libraries, notebooks, and variables. Here is how to handle each.

Pattern 1: Shared Python Libraries as Wheel Artifacts

Place shared library code in a top-level shared_libraries folder. In each bundle's databricks.yml, declare a Python wheel artifact with type set to whl, path pointing to ../shared_libraries, and a build command of "pip install build and python -m build". Bundles reference the built wheel as a cluster library. This avoids duplicating library code and ensures all bundles use the same version.

Pattern 2: Shared Notebooks via Relative Path References

Put utility notebooks in a top-level shared_notebooks folder. Reference them from bundle notebooks using relative paths like ../../shared_notebooks/transform_utils. Never copy notebooks between bundles — a single source of truth prevents version drift. This is the simplest pattern to adopt first.

Pattern 3: Shared Variables via Root Include

Define environment-agnostic variables (catalog name, service account, common tags) in a root-level variables.yml. In each bundle's databricks.yml, add an include directive pointing to ../variables.yml. Bundle-specific overrides go in the bundle's own variable files. This keeps environment configuration consistent without duplication.

When to Split a Domain Out

Even if you start monorepo, watch for these signals that it is time to split a domain out:

• That domain's CI/CD changes no longer benefit from shared libraries
• The team wants a different deployment cadence from the rest
• Access control requests for that domain keep escalating
• That domain's bundle tests account for more than 40% of total CI time

Splitting one domain out of a monorepo is a tractable migration. Consolidating multiple repos into one later is much harder. So if you are unsure, start monorepo and split intentionally when the signal is clear.

Happy to discuss any of these patterns further — especially the wheel artifact build pipeline and CI path-filtering setup, which tend to be where teams get stuck first.