Mono Repo or not Mono Repo - that is the question?

[MrDecisive]

Hi All,

We are just getting started with a Databricks on Azure Proof of Concept platform. Right now we are thinking about the deployment of workloads using data bundles driven by GitHub Action workflows. The documents recommend using a single repo for all data bundles and managing some level of separation with folders and code owner files. Our concern comes from the idea of having multiple unrelated groups pushing and pulling from the same repo - and all the potential risks that poses. The alternative is to have multiple repos - one per team - but our searched suggest this adds a lot of administration overhead and duplication or code resulting in a more complex maintenance profile. We are wondering what people do out in the real world who are using databricks in large organisations? Any thoughts or suggestions will be gratefully received.

Cheers

[DoTA]

We run this at a mid-size scale (multiple business units on one Databricks-on-Azure workspace set) and landed on a hybrid, which I'd recommend over either pure extreme:

- One repo per team/domain for the actual pipeline code (bundle resources + src). This gives natural blast-radius isolation, independent CI/CD pipelines, and CODEOWNERS enforcement without fighting folder-level permissions inside a single shared repo.

- A shared/platform repo for common bundle templates, reusable Python packages (utils, connectors, logging wrappers), and CI workflow templates that team repos pull in via a published internal package or reusable workflow. This is what kills the "duplication" pain people cite as the monorepo's advantage - you're not duplicating logic, you're just not co-locating it in the same git history as unrelated teams.

- Environment/target definitions (dev/staging/prod, cluster policies, service principals) live in the platform repo too, referenced by each team repo's databricks.yml via variable overrides, so the platform team retains control of where things deploy without touching each team's pipeline code.

Why not the single-repo-with-folders approach the docs suggest: at any real headcount, "separation with folders and code owner files" degrades badly. Branch protection rules, PR review requirements, and CI triggers all get harder to scope precisely per-folder, and one team's bad merge/rebase can block CI for everyone. It works fine at PoC scale, which is where you are now, but it's worth revisiting before you onboard a 2nd or 3rd team.

One thing multi-repo genuinely costs you: cross-team bundle dependencies (Team A's pipeline needs Team B's Delta table) still need a contract. We handle that with Unity Catalog grants plus a lightweight schema registry doc, not git.

[balajij8 · Accepted solution]

Hi MrDecisive,

You can go with a Mono repo governed by Path Filtering and code owner approach for PoC & early scaling phases. The administrative overhead of setting up and managing dozens of repositories during a PoC will slow down team's velocity. It's the recommended approach with low initial CI/CD Overhead. You configure unified pipelines, centralized secrets and reusable workflow templates once. Teams can easily share common Python utilities, base bundle configurations, and custom data quality functions without managing separate package feeds. It relies on Git controls such as directory ownership rules and path filtering to isolate environments and changes. You can setup proper repository governance rather than splitting code across multiple repositories.

Folder Layout - 

You can structure the single repository with clear boundaries. Dedicated workflows directory to handle deployment definitions. Central shared folder for common libraries, reusable helper scripts, and base bundle templates maintained by the core platform team. Top-level bundles directory contains distinct subfolders for each team ( bundles/marketing_analytics bundles/finance_forecasting) holding its own databricks.yml file and resources.

Setup Considerations

• Path-Filtered Deployment Workflows - Configure GitHub Actions using path filters so that a push to a specific team folder only triggers the deployment pipeline for that specific bundle. A broken deployment in the finance folder will never block or affect the marketing team's pipeline.
• CODE OWNERS Rules - Define explicit ownership paths so that developers cannot merge changes into directories they don't own without an explicit approval from that specific team's code owners. Changes to the shared platform folder should require approval from the core platform team.
• Branch Protection Enforcement - Lock down the main branch to prevent direct pushes. Require pull requests to pass automated validation checks (databricks bundle validate, linting and tests) alongside mandatory approvals from designated code owners before any merge can occur.

It gives the simplicity you need in a PoC while establishing a modular structure that can easily be split into domain-level mono repos later if you need it. Use the alternative approach of multi repos only if you have hard isolation requirements (legal etc)

More details here

[rkhand14_ltm]

@MrDecisive 

Based on your requirements, I would recommend adopting a dedicated repository per team. While it may require some upfront effort in terms of setup and governance, it will significantly streamline development and deployment processes in the long run. Decoupling teams and codebases improves autonomy, reduces dependencies, and simplifies maintenance.

A single repository is more suitable in a Data Fabric architecture where multiple teams contribute shared code, definitions, or assets that need to be centrally managed and consumed by downstream systems. In such scenarios, a fork-based workflow may be more practical than relying solely on pull requests.

Overall, unless you are implementing a Data Fabric model or building a quick proof of concept, a repository-per-team approach aligns better with Data Mesh principles and is generally the preferred option.

[MrDecisive]

Thanks for all the replies so far - definitely food for thought. I think as this is a POC we will probably go with the mono repo for easy of deployment with the view that as we learn more about our potential customer landscape we can always pivot to separate repos or a hybrid approach as best suits our position. Again thanks for all the really good information. Watch this space!