Bizarre Delta Tables pipeline error: ModuleNotFound

Geoff — Tue, 12 Dec 2023 21:33:05 GMT

I received the following error when trying to import a function defined in a .py file into a .ipynb file. I would add code blocks, but the message keeps getting rejected for invalid HTML.

# test_lib.py (same directory, in a subfolder)
def square(x):
return x * x

# dlt_pipeline.ipynb
import dlt
import sys
import os
from pathlib import Path
sys.path.append(os.path.abspath('..'))
from pyspark.sql.functions import expr
from test_lib import square # Error is here

ModuleNotFoundError: No module named 'test_lib',None,Map(),Map(),List(),List(),Map())

I was using Asset Bundles when this occurred. The error makes no sense to me because (1) I can import that function when I run the .ipynb file locally, or even when I manually run it on DB using my personal cluster, and (2) the ',None,Map(),Map(),List(),List(),Map()' at the end seems to come from nowhere. Any help in solving this would be very much appreciated. Thank you!

topic Bizarre Delta Tables pipeline error: ModuleNotFound in Data Engineering

Bizarre Delta Tables pipeline error: ModuleNotFound