topic Re: DLT - Importing Python Package in Data Engineering

DLT - Importing Python Package

YS1 — Wed, 07 Aug 2024 23:05:34 GMT

Hello,

I'm creating a DLT pipeline where I read a Kafka stream, perform transformations using UDFs, and save the data in multiple tables. When I define the functions directly in the same notebook, the code works fine. However, if I move the code into a Python file and try to import it as a package, the pipeline fails with the following error:

I've tried renaming the files and changing their locations -although the pipeline and the package are in the same directory-

Despite these efforts, nothing worked. Here is how I'm trying to import the module:

Any insights or solutions would be greatly appreciated!

Re: DLT - Importing Python Package

szymon_dybczak — Thu, 08 Aug 2024 12:01:11 GMT

Hi @YS1 ,

Have you added the python file in the Pipeline settings, in the list of source code?

Re: DLT - Importing Python Package

YS1 — Thu, 08 Aug 2024 17:16:50 GMT

yes, I did, the pipeline would work for seconds and then fails with the same error "No module named 'eventlog_utils'". And that's why it's confusing because I tried this on another pipeline with different setup/functions and it works fine!