Databricks Community

a_user12 · Saturday

Hi!

I create a bundle with

databricks bundle init

and i let create a pipeline including a package.

The pipeline definition looks like the following

# The main pipeline for pipelinewithpackage

resources:
  pipelines:
    pipelinewithpackage_etl:
      name: pipelinewithpackage_etl
      catalog: ${var.catalog}
      schema: ${var.schema}
      serverless: true
      root_path: "../src/pipelinewithpackage_etl"

      libraries:
        - glob:
            include: ../src/pipelinewithpackage_etl/transformations/**

      environment:
        dependencies:
          # We include every dependency defined by pyproject.toml by defining an editable environment
          # that points to the folder where pyproject.toml is deployed.
          - --editable ${workspace.file_path}

The package looks like the following

Now, I want to use the package in my pipeline. So I simply added

from pipelinewithpackage import taxis

before my @DP.table defintions. This is the only modificaiton which I did - the rest of the code is the generatedbundle code.

I deploy the bundle and run it: then i get the ModuleNotFoundError: No module named 'pipelinewithpackage' error.

In the cluster logs I can see, that the wheel is created:

Building wheels for collected packages: pipelinewithpackage
  Building editable for pipelinewithpackage (pyproject.toml): started
  Building editable for pipelinewithpackage (pyproject.toml): finished with status 'done'
  Created wheel for pipelinewithpackage: filename=pipelinewithpackage-0.0.1-py3-none-any.whl size=1433 sha256=492c44e0c2d9eb063b3f3c01a6fffe9d1699de5123a389a0e8b5708bbf61060c
Successfully built pipelinewithpackage
Installing collected packages: pipelinewithpackage
Successfully installed pipelinewithpackage-0.0.1

Why isn't it finding the library?

Thank you!

Louis_Frolio · yesterday

Hello @a_user12, .

Your bundle config is the documented pattern for pulling pyproject.toml dependencies into a pipeline, so the setup itself is sound. The clue is the 1,433-byte wheel. An editable install doesn't copy your code into site-packages. It installs a small pointer telling Python which folder to add to sys.path. So "Successfully installed" only means the pointer landed. If it points at the wrong folder, the import fails exactly the way you're seeing.

Here's what I'd check:

Make sure pyproject.toml points hatchling at the real package folder. Your layout should look like this:

src/
  pipelinewithpackage/
    __init__.py
    taxis.py

[tool.hatch.build.targets.wheel]
packages = ["src/pipelinewithpackage"]

Prove the packaging works by building a regular (non-editable) wheel locally with uv build --wheel, then run unzip -l dist/*.whl. You should see pipelinewithpackage/__init__.py and pipelinewithpackage/taxis.py. If you only see *.dist-info files, the config in step 1 is the culprit.
Confirm pyproject.toml and src/pipelinewithpackage/ actually deployed under the path ${workspace.file_path} resolves to. The libraries.glob entry only covers your transformation files. The editable install has to find the project metadata and the package folder on its own.

If you'd rather skip packaging, you have two options. You can put the helper under the pipeline's utilities folder and import it directly. Or you can pass the source path in and add it to sys.path yourself:

configuration:
  bundle.sourcePath: ${workspace.file_path}/src

import sys
sys.path.append(spark.conf.get("bundle.sourcePath", "."))
from pipelinewithpackage import taxis

One caution: don't try to fix this with %pip install or dbutils.library.restartPython() inside the pipeline. Pipelines don't support either, and dependencies belong in the Environment settings.

If it's still stubborn, paste your pyproject.toml and folder tree as text and I'll take another look.

References:

Bundle library dependencies: https://docs.databricks.com/aws/en/dev-tools/bundles/library-dependencies
Manage Python dependencies for pipelines: https://docs.databricks.com/aws/en/ldp/developer/external-dependencies
Import Python modules from Git folders or workspace files: https://docs.databricks.com/aws/en/ldp/import-workspace-files
Develop pipelines with Declarative Automation Bundles: https://docs.databricks.com/aws/en/dev-tools/bundles/pipelines-tutorial

Hope this helps.

Regards, Louis.

a_user12 · yesterday

Hi - Thank you for your quick response.

I checked 1-3: everything seems to be fine:

I can build manually and see all files
```
workspace.file_path
```

contains the files

So I provide additional information.

Folder Tree

ª   .gitignore
ª   AGENTS.md
ª   CLAUDE.md
ª   databricks.yml
ª   pyproject.toml
ª   README.md
ª   uv.lock
+---fixtures
ª       .gitkeep
ª       
+---resources
ª       pipelinewithpackage_etl.pipeline.yml
ª       
+---src
ª   ª   sample_notebook.ipynb
ª   ª   
ª   +---pipelinewithpackage
ª   ª       main.py
ª   ª       taxis.py
ª   ª       __init__.py
ª   ª       
ª   +---pipelinewithpackage_etl
ª       ª   README.md
ª       ª   
ª       +---explorations
ª       ª       sample_exploration.ipynb
ª       ª       
ª       +---transformations
ª               sample_trips_pipelinewithpackage.py
ª               sample_zones_pipelinewithpackage.py
ª               
+---tests
        conftest.py
        sample_taxis_test.py

My Pipeline code (where I get the error):

from pyspark import pipelines as dp
from pyspark.sql.functions import col

from pipelinewithpackage import taxis
# This file defines a sample transformation.
# Edit the sample below or add new transformations
# using "+ Add" in the file browser.


@dp.table
def sample_trips_pipelinewithpackage():
    return spark.read.table("samples.nyctaxi.trips")

Pyproject.toml

[project]
name = "pipelinewithpackage"
version = "0.0.1"
authors = [{ name = "xxxx" }]
requires-python = ">=3.10,<3.13"
dependencies = [
    # Any dependencies for jobs and pipelines in this project can be added here
    # See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
    #
    # LIMITATION: for pipelines, dependencies are cached during development;
    # add dependencies to the 'environment' section of your pipeline.yml file instead
]

[dependency-groups]
dev = [
    "pytest",
    "ruff",
    "databricks-dlt",
    "databricks-connect>=15.4,<15.5",
    "ipykernel",
]

[tool.hatch.build.targets.wheel]
packages = ["src/pipelinewithpackage"]

[project.scripts]
main = "pipelinewithpackage.main:main"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.ruff]
line-length = 120

Just FYI:
This is, how it looks in my settings