topic Re: Asset Bundles: Shared libraries and notebooks in monorepo multi-bundle setup in Data Engineering

Asset Bundles: Shared libraries and notebooks in monorepo multi-bundle setup

auso — Thu, 15 May 2025 14:53:48 GMT

I am part of a small team of Data Engineers which started using Databricks Asset Bundles one year ago. Our code base consists of typical ETL-workloads written primarily in Jupyter notebooks (.ipynb), and jobs (.yaml) with our codebase spanning across a large number of different business domains.

Currently, our code base consists of a single monorepo with one large bundle containing all our notebooks, jobs, libraries etc.

Our code base has grown to a size where we see the need to split our single bundle into several smaller bundles - one for each business domain.

We are envisioning a setup similar to the following (simplified) structure:

monorepo/ │ ├── shared_notebooks/ ├── shared_libraries/ ├── variables.yml │ ├── Bundle_A/ │ ├── resources/ │ ├── src/ │ └── databricks.yml │ └── Bundle_B/ ├── resources/ ├── src/ └── databricks.yml

Where the repo contains some shared notebooks and libraries which may be used in all bundles in our repository.

Does anyone have some suggestions for how this should be implemented?

How can we "import" shared assets (notebooks, libraries and variables) into our bundles?
Does our approach to splitting up our mono-bundle repository seem sensible?

Thanks in advance for your insights!

Kaspar Hauser

Re: Asset Bundles: Shared libraries and notebooks in monorepo multi-bundle setup

bee-jugger — Tue, 02 Sep 2025 11:15:20 GMT

@auso , did you get an answer or a solution?

Re: Asset Bundles: Shared libraries and notebooks in monorepo multi-bundle setup

Witold — Tue, 02 Sep 2025 12:32:24 GMT

Yes, it's feasible. In DAB you just need to use paths to import common libraries.

Re: Asset Bundles: Shared libraries and notebooks in monorepo multi-bundle setup

-werners- — Tue, 02 Sep 2025 12:53:08 GMT

1. the easiest way to do this is to package your shared librabries into a wheel (suppose you use python). Like that you do not have to mess with the pythonpath and you can install these libs automatically to any cluster (via policies or dabs or whatever).

2. totally makes sense, we do it in a similar way

Re: Asset Bundles: Shared libraries and notebooks in monorepo multi-bundle setup

DoTA — Sat, 08 Aug 2026 03:11:55 GMT

Your proposed structure is sensible — it is the standard pattern for large-team monorepo setups with Databricks Asset Bundles. Here is a more complete answer to both questions you raised.

1. Shared Python libraries: build a wheel, wire it as a bundle artifact

Package shared_libraries/ as a standard Python wheel (setup.py or pyproject.toml). In each bundle's databricks.yml, declare it as an artifact with a relative path pointing up to the shared directory. Set a build command under the target so DABs builds the wheel during "databricks bundle deploy". It will then install it on cluster automatically via the bundle's library configuration — no %pip install needed in notebooks.

The key configuration: set "type: whl", "path: ../shared_libraries", and "build: pip install build && python -m build" under the target. DABs resolves the path relative to the bundle's databricks.yml file, so ../shared_libraries correctly reaches the sibling directory at the monorepo root.

2. Shared notebooks: use relative path references, never copy

DABs resolves notebook_path values relative to the databricks.yml that declares the job. For notebooks that live above the bundle root, use "../" traversal: "notebook_path: ../../shared_notebooks/transform_utils". This works for any depth as long as the target path stays within the repository root.

The critical rule: do not copy shared notebooks into each bundle's src/ directory. Copies drift immediately once the shared notebook is updated and you forget to propagate the change.

3. Shared variables: root-level variables.yml with include

Place a variables.yml at the monorepo root with catalog_name, environment prefix, and any other cross-bundle constants. In each bundle's databricks.yml, add an "include: [../variables.yml]" block. Bundle-level variables with the same key override root-level ones, so you get shared defaults with clean per-bundle overrides where needed.

On whether splitting one large bundle into domain bundles is the right call

Yes, for your situation it is. A monorepo with multiple bundles is the right pattern when teams share the same CI pipeline and deployment cadence, shared library code changes infrequently or is versioned via the wheel, and you want cross-bundle dependency visibility in a single repository.

The only reason to split into separate repositories is if business domains have genuinely different release cycles (one team ships 10 times a day, another ships weekly), if repository-level access control is required for compliance, or if the CI pipeline becomes too slow even with path-based change filtering. For most teams under roughly 30 engineers and 5 business domains, the monorepo structure you described handles this cleanly.