cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT - Importing Python Package

YS1
Contributor

Hello,

I'm creating a DLT pipeline where I read a Kafka stream, perform transformations using UDFs, and save the data in multiple tables. When I define the functions directly in the same notebook, the code works fine. However, if I move the code into a Python file and try to import it as a package, the pipeline fails with the following error:

YS1_1-1723071739598.png

I've tried renaming the files and changing their locations -although the pipeline and the package are in the same directory-

Despite these efforts, nothing worked. Here is how I'm trying to import the module:

YS1_0-1723071683421.png

Any insights or solutions would be greatly appreciated!

2 REPLIES 2

szymon_dybczak
Contributor III

Hi @YS1 ,

Have you added the python file in the Pipeline settings, in the list of source code? 

Slash_1-1723118449935.png

 

 

 

 

yes, I did, the pipeline would work for seconds and then fails with the same error "No module named 'eventlog_utils'". And that's why it's confusing because I tried this on another pipeline with different setup/functions and it works fine!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group