I am following the Databrick's documentation on unit testing found here: Run tests with pytest for the Databricks extension for Visual Studio Code - Azure Databricks | Micro...
However, when taking it a step further I get an ImportError or a ModuleNotFoundError, when attempting a relative and absolute import of a sibling package, respectively.
My current folder structure is:
myinvoke.py src __init__.py myfunctionfile.py tests test_myfunction_test.py
test_myfunction_test.py unit tests myfunction with the follow import: from src.myfunctionfile import myfunction. I have tried both relative and absolute imports for this with no result.
When I run the test as per the tutorial in the link above,through the VSCode Databricks extension, it runs myinvoke.py by invoking pytest.main(), as expected, but pytest gets interrupted when collecting the test_myfunction_test function tests with ImportError: attempted relative import beyond top-level package.
These errors don't occur when running pytest locally. Why does the cluster env not recognize these packages?
Import errors happen often with Pytest. To Debug this error you can add this in your "test_myfunction_test.py":
import sys # printing all directories for # interpreter to search sys.path
sys.path is a built-in variable within the sys module. It contains a list of directories that the interpreter will search in for the required module
In order to do from src.myfunctionfile, your project directory needs to be in the list of "sys.path" paths
A quick fix would be to use
Otherwise you should check how your PYTHONPATH variable is set or your pytest.ini file
Hope this helps 🙂
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!