cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Declarative Pipeline with editable wheel: PYTHON.MODULE_NOT_FOUND_ERROR

a_user12
Contributor

Hi!

I create a bundle with 

databricks bundle init

and i let create a pipeline including a package. 

The pipeline definition looks like the following

# The main pipeline for pipelinewithpackage

resources:
  pipelines:
    pipelinewithpackage_etl:
      name: pipelinewithpackage_etl
      catalog: ${var.catalog}
      schema: ${var.schema}
      serverless: true
      root_path: "../src/pipelinewithpackage_etl"

      libraries:
        - glob:
            include: ../src/pipelinewithpackage_etl/transformations/**

      environment:
        dependencies:
          # We include every dependency defined by pyproject.toml by defining an editable environment
          # that points to the folder where pyproject.toml is deployed.
          - --editable ${workspace.file_path}

The package looks like the following

databricks.png

Now, I want to use the package in my pipeline. So I simply added 

from pipelinewithpackage import taxis

before my @DP.table defintions. This is the only modificaiton which I did - the rest of the code is the generatedbundle code.

I deploy the bundle and run it: then i get the ModuleNotFoundError: No module named 'pipelinewithpackage' error.

In the cluster logs I can see, that the wheel is created:

Building wheels for collected packages: pipelinewithpackage
  Building editable for pipelinewithpackage (pyproject.toml): started
  Building editable for pipelinewithpackage (pyproject.toml): finished with status 'done'
  Created wheel for pipelinewithpackage: filename=pipelinewithpackage-0.0.1-py3-none-any.whl size=1433 sha256=492c44e0c2d9eb063b3f3c01a6fffe9d1699de5123a389a0e8b5708bbf61060c
Successfully built pipelinewithpackage
Installing collected packages: pipelinewithpackage
Successfully installed pipelinewithpackage-0.0.1

Why isn't it finding the library?

Thank you!

 

 

0 REPLIES 0