cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Pytest imports of sibling modules when using Databricks for VSCode

TaBorjaTa
New Contributor II

Hello all, 

I am following the Databrick's documentation on unit testing found here: Run tests with pytest for the Databricks extension for Visual Studio Code - Azure Databricks | Micro...

However, when taking it a step further I get an ImportError or a ModuleNotFoundError, when attempting a relative and absolute import of a sibling package, respectively. 

My current folder structure is: 

.
myinvoke.py src __init__.py myfunctionfile.py tests test_myfunction_test.py

 test_myfunction_test.py unit tests myfunction with the follow import: from src.myfunctionfile import myfunction. I have tried both relative and absolute imports for this with no result. 

When I run the test as per the tutorial in the link above,through the VSCode Databricks extension, it runs myinvoke.py by invoking pytest.main(), as expected, but pytest gets interrupted when collecting the test_myfunction_test function tests with ImportError: attempted relative import beyond top-level package.

These errors don't occur when running pytest locally. Why does the cluster env not recognize these packages?

Thank you!

1 REPLY 1

Trifa
New Contributor II

Hello

Import errors happen often with Pytest. To Debug this error you can add this in your "test_myfunction_test.py":

import sys 
# printing all directories for  
# interpreter to search 
sys.path

sys.path is a built-in variable within the sys module. It contains a list of directories that the interpreter will search in for the required module

In order to do from src.myfunctionfile, your project directory needs to be in the list of "sys.path" paths 

A quick fix would be to use

sys.path.append("/you/project/directory") 

Otherwise you should check how your PYTHONPATH variable is set or your pytest.ini file

Hope this helps 🙂

 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!