Bizarre Delta Tables pipeline error: ModuleNotFound

[Geoff]

I received the following error when trying to import a function defined in a .py file into a .ipynb file. I would add code blocks, but the message keeps getting rejected for invalid HTML.

# test_lib.py (same directory, in a subfolder)
def square(x):
return x * x

# dlt_pipeline.ipynb
import dlt
import sys
import os
from pathlib import Path
sys.path.append(os.path.abspath('..'))
from pyspark.sql.functions import expr
from test_lib import square # Error is here

ModuleNotFoundError: No module named 'test_lib',None,Map(),Map(),List(),List(),Map())

I was using Asset Bundles when this occurred. The error makes no sense to me because (1) I can import that function when I run the .ipynb file locally, or even when I manually run it on DB using my personal cluster, and (2) the ',None,Map(),Map(),List(),List(),Map()' at the end seems to come from nowhere. Any help in solving this would be very much appreciated. Thank you!

[Kaniz]

Hi @Geoff, The error message ModuleNotFoundError: No module named 'test_lib' indicates that Python cannot find the module test_lib. This could be due to several reasons:

1. File Location: The Python file test_lib.py needs to be in the same directory as your Jupyter notebook or in a directory that’s included in sys.path. You mentioned that test_lib.py is in a subfolder. If that’s the case, you need to include the subfolder in the path when importing the module. For example, if test_lib.py is in a subfolder named subfolder, you would import it like this:

from subfolder.test_lib import square

1. Jupyter Kernel: The Jupyter Notebook might be using a different Python kernel, which could have a different sys.path. You can check the Python executable being used by the Jupyter kernel with the following code:

import sys print(sys.executable)

1. Asset Bundles: If you’re using Databricks Asset Bundles, there might be some specific configurations needed. I recommend checking the Databricks Asset Bundles documentation for more details.

The part of the error message with None,Map(),Map(),List(),List(),Map() seems unrelated to the ModuleNotFoundError. It might be part of a different error message or log, possibly related to the Asset Bundles.

Remember to replace 'subfolder' with the actual name of your subfolder. If these suggestions don’t solve your issue, could you provide more details about your folder structure and how you’re running the notebook? That would help in giving more specific advice.