Pytest imports of sibling modules when using Databricks for VSCode

TaBorjaTa — Thu, 09 Nov 2023 15:42:23 GMT

Hello all,

I am following the Databrick's documentation on unit testing found here: Run tests with pytest for the Databricks extension for Visual Studio Code - Azure Databricks | Microsoft Learn

However, when taking it a step further I get an ImportError or a ModuleNotFoundError, when attempting a relative and absolute import of a sibling package, respectively.

My current folder structure is:

.
myinvoke.py
    src
        __init__.py
        myfunctionfile.py 
    tests
        test_myfunction_test.py

test_myfunction_test.py unit tests myfunction with the follow import: from src.myfunctionfile import myfunction. I have tried both relative and absolute imports for this with no result.

When I run the test as per the tutorial in the link above,through the VSCode Databricks extension, it runs myinvoke.py by invoking pytest.main(), as expected, but pytest gets interrupted when collecting the test_myfunction_test function tests with ImportError: attempted relative import beyond top-level package.

These errors don't occur when running pytest locally. Why does the cluster env not recognize these packages?

Thank you!

Re: Pytest imports of sibling modules when using Databricks for VSCode

Trifa — Wed, 22 Nov 2023 09:00:38 GMT

Hello

Import errors happen often with Pytest. To Debug this error you can add this in your "test_myfunction_test.py":

import sys # printing all directories for # interpreter to search sys.path

sys.path is a built-in variable within the sys module. It contains a list of directories that the interpreter will search in for the required module

In order to do from src.myfunctionfile, your project directory needs to be in the list of "sys.path" paths

A quick fix would be to use

sys.path.append("/you/project/directory")

Otherwise you should check how your PYTHONPATH variable is set or your pytest.ini file

Hope this helps 🙂

topic Re: Pytest imports of sibling modules when using Databricks for VSCode in Data Engineering

Pytest imports of sibling modules when using Databricks for VSCode

Re: Pytest imports of sibling modules when using Databricks for VSCode