Declarative Pipeline with editable wheel: PYTHON.MODULE_NOT_FOUND_ERROR

[a_user12]

Hi!

I create a bundle with 

databricks bundle init

and i let create a pipeline including a package. 

The pipeline definition looks like the following

# The main pipeline for pipelinewithpackage

resources:
  pipelines:
    pipelinewithpackage_etl:
      name: pipelinewithpackage_etl
      catalog: ${var.catalog}
      schema: ${var.schema}
      serverless: true
      root_path: "../src/pipelinewithpackage_etl"

      libraries:
        - glob:
            include: ../src/pipelinewithpackage_etl/transformations/**

      environment:
        dependencies:
          # We include every dependency defined by pyproject.toml by defining an editable environment
          # that points to the folder where pyproject.toml is deployed.
          - --editable ${workspace.file_path}

The package looks like the following

Now, I want to use the package in my pipeline. So I simply added 

from pipelinewithpackage import taxis

before my @DP.table defintions. This is the only modificaiton which I did - the rest of the code is the generatedbundle code.

I deploy the bundle and run it: then i get the ModuleNotFoundError: No module named 'pipelinewithpackage' error.

In the cluster logs I can see, that the wheel is created:

Building wheels for collected packages: pipelinewithpackage
  Building editable for pipelinewithpackage (pyproject.toml): started
  Building editable for pipelinewithpackage (pyproject.toml): finished with status 'done'
  Created wheel for pipelinewithpackage: filename=pipelinewithpackage-0.0.1-py3-none-any.whl size=1433 sha256=492c44e0c2d9eb063b3f3c01a6fffe9d1699de5123a389a0e8b5708bbf61060c
Successfully built pipelinewithpackage
Installing collected packages: pipelinewithpackage
Successfully installed pipelinewithpackage-0.0.1

Why isn't it finding the library?

Thank you!

[Louis_Frolio]

Hello @a_user12, .

Your bundle config is the documented pattern for pulling pyproject.toml dependencies into a pipeline, so the setup itself is sound. The clue is the 1,433-byte wheel. An editable install doesn't copy your code into site-packages. It installs a small pointer telling Python which folder to add to sys.path. So "Successfully installed" only means the pointer landed. If it points at the wrong folder, the import fails exactly the way you're seeing.

Here's what I'd check:

1. Make sure pyproject.toml points hatchling at the real package folder. Your layout should look like this:

src/
  pipelinewithpackage/
    __init__.py
    taxis.py

[tool.hatch.build.targets.wheel]
packages = ["src/pipelinewithpackage"]

2. Prove the packaging works by building a regular (non-editable) wheel locally with uv build --wheel, then run unzip -l dist/*.whl. You should see pipelinewithpackage/__init__.py and pipelinewithpackage/taxis.py. If you only see *.dist-info files, the config in step 1 is the culprit.

3. Confirm pyproject.toml and src/pipelinewithpackage/ actually deployed under the path ${workspace.file_path} resolves to. The libraries.glob entry only covers your transformation files. The editable install has to find the project metadata and the package folder on its own.

If you'd rather skip packaging, you have two options. You can put the helper under the pipeline's utilities folder and import it directly. Or you can pass the source path in and add it to sys.path yourself:

configuration:
  bundle.sourcePath: ${workspace.file_path}/src

import sys
sys.path.append(spark.conf.get("bundle.sourcePath", "."))
from pipelinewithpackage import taxis

One caution: don't try to fix this with %pip install or dbutils.library.restartPython() inside the pipeline. Pipelines don't support either, and dependencies belong in the Environment settings.

If it's still stubborn, paste your pyproject.toml and folder tree as text and I'll take another look.

References:

• Bundle library dependencies: https://docs.databricks.com/aws/en/dev-tools/bundles/library-dependencies
• Manage Python dependencies for pipelines: https://docs.databricks.com/aws/en/ldp/developer/external-dependencies
• Import Python modules from Git folders or workspace files: https://docs.databricks.com/aws/en/ldp/import-workspace-files
• Develop pipelines with Declarative Automation Bundles: https://docs.databricks.com/aws/en/dev-tools/bundles/pipelines-tutorial

Hope this helps.

Regards, Louis.

[a_user12]

Hi - Thank you for your quick response.

I checked 1-3: everything seems to be fine:

• I can build manually and see all files
•  

  workspace.file_path​

contains the files

So I provide additional information.

Folder Tree

ª   .gitignore
ª   AGENTS.md
ª   CLAUDE.md
ª   databricks.yml
ª   pyproject.toml
ª   README.md
ª   uv.lock
+---fixtures
ª       .gitkeep
ª       
+---resources
ª       pipelinewithpackage_etl.pipeline.yml
ª       
+---src
ª   ª   sample_notebook.ipynb
ª   ª   
ª   +---pipelinewithpackage
ª   ª       main.py
ª   ª       taxis.py
ª   ª       __init__.py
ª   ª       
ª   +---pipelinewithpackage_etl
ª       ª   README.md
ª       ª   
ª       +---explorations
ª       ª       sample_exploration.ipynb
ª       ª       
ª       +---transformations
ª               sample_trips_pipelinewithpackage.py
ª               sample_zones_pipelinewithpackage.py
ª               
+---tests
        conftest.py
        sample_taxis_test.py
        

My Pipeline code (where I get the error):

from pyspark import pipelines as dp
from pyspark.sql.functions import col

from pipelinewithpackage import taxis
# This file defines a sample transformation.
# Edit the sample below or add new transformations
# using "+ Add" in the file browser.


@dp.table
def sample_trips_pipelinewithpackage():
    return spark.read.table("samples.nyctaxi.trips")

Pyproject.toml

[project]
name = "pipelinewithpackage"
version = "0.0.1"
authors = [{ name = "xxxx" }]
requires-python = ">=3.10,<3.13"
dependencies = [
    # Any dependencies for jobs and pipelines in this project can be added here
    # See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
    #
    # LIMITATION: for pipelines, dependencies are cached during development;
    # add dependencies to the 'environment' section of your pipeline.yml file instead
]

[dependency-groups]
dev = [
    "pytest",
    "ruff",
    "databricks-dlt",
    "databricks-connect>=15.4,<15.5",
    "ipykernel",
]

[tool.hatch.build.targets.wheel]
packages = ["src/pipelinewithpackage"]

[project.scripts]
main = "pipelinewithpackage.main:main"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.ruff]
line-length = 120

Just FYI:
This is, how it looks in my settings

[Louis_Frolio · Accepted solution]

Hello @a_user12, 

Thanks for running those checks and posting the tree and pyproject.toml. That rules out packaging. Your layout and Hatch config are correct, and the files deployed where they should.

That points to the runtime side. An editable install drops a small .pth file into site-packages, and that file tells Python to add .../files/src to sys.path. If the pipeline sets up its environment in a way that skips .pth processing, pip reports success and the import still fails. I can't prove that from the thread alone, so let's have the pipeline tell us. Temporarily swap your import for this:

import sys
import importlib.metadata as md

try:
    from pipelinewithpackage import taxis
except ModuleNotFoundError:
    dist = md.distribution("pipelinewithpackage")
    pth = [f"{f}: {f.read_text()}" for f in dist.files if str(f).endswith(".pth")]
    raise RuntimeError(f"PTH: {pth}\nSYS.PATH: {sys.path}")

The details will show up in the event log. Here's how to read them:

• If the .pth points at your src folder but that path isn't in sys.path, the pointer never got applied.
• If md.distribution fails, the package was installed into a different environment than the one running your code.

Either way, here are a few ways to get unblocked.

The quickest is to add the source folder to the path yourself. Keep your existing --editable dependency and add a configuration block to the pipeline:

configuration:
  bundle.sourcePath: ${workspace.file_path}/src

Then read that value in your transformation file before the import:

import sys
sys.path.append(spark.conf.get("bundle.sourcePath"))
from pipelinewithpackage import taxis

The config value does nothing by itself. It's the sys.path.append that does the work. Also, ${workspace.file_path} already resolves to a full /Workspace/... path, so don't add another prefix.

The most deterministic option is to drop the editable install and build a regular wheel through bundle artifacts (build: uv build --wheel), then reference the built wheel in the pipeline environment. That removes the .pth from the picture entirely. Check the library dependencies page for the exact wheel path form your CLI version supports.

If taxis is only used by this pipeline, you can also move it into the pipeline's utilities folder. The pipeline root is added to sys.path automatically.

If the diagnostic confirms the .pth isn't being applied, please open a support ticket. The template ships with this exact pattern, so it should work out of the box.

References:

• Import Python modules from Git folders or workspace files: https://docs.databricks.com/aws/en/ldp/import-workspace-files
• Bundle library dependencies: https://docs.databricks.com/aws/en/dev-tools/bundles/library-dependencies
• Bundle configuration examples: https://docs.databricks.com/aws/en/dev-tools/bundles/examples
• Manage Python dependencies for pipelines: https://docs.databricks.com/aws/en/ldp/developer/external-dependencies
• pip editable installs: https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs

Regards, Louis.

[a_user12]

Thank you @Louis_Frolio !

You diagnosis is correct (the src folder is not part of the sys.path folder). I have to say, that the databricks documentation here https://docs.databricks.com/aws/en/dev-tools/bundles/library-dependencies#uv-and-pyprojecttoml implies that.

Further, also you mitigation works!

Thank you for your effort!