โ05-20-2024 03:55 AM
Dear all,
I am following the guide in this article: https://docs.databricks.com/en/notebooks/testing.html
however I am unable to run pytest due to the following error: ImportError while importing test module '/Workspace/Users/deadmanhide@gmail.com/test_trans.py'
and
E ModuleNotFoundError: No module named 'test_trans'
My setup is:
Workspace:
run_tests.py (NoteBook where I install pytest and run pytest.main)
test_trans.py(Python file containing the unit tests)
transform (folder)
-- operations.py(Notebook with transform and cleansing functions)
-- __init__.py
I also tried to put test file and transform file inside the folders with __init__.py file so it would be treated as a package, also tried to do the same in Repos and not on workspace.
Clearly I am doing something wrong, would greatly appreciate any help,
Kind regards
โ05-22-2024 10:47 AM
Thank you very much for a detailed answer Kaniz! I have followed the steps you described and decided to simplify the structure further, now I have all 3 files at the same level and now I can collect my 2 tests.
The issue is with the module imports again, I have followed the guide from DataBricks regarding Unit Testing howecver it seems I am still missing something.
In my workspace I have two notebooks (transform_functions and run_tests) and a python file test_trans.py file, all at the same level.
The error I am getting is:
FAILED test_trans.py::test_check_columns_exist - ModuleNotFoundError: No module named 'transform_functions' FAILED test_trans.py::test_transform_replace_nulls - ModuleNotFoundError: No module named 'transform_functions'
def test_check_columns_exist(get_sparksession, get_test_df): > from transform_functions import * E ModuleNotFoundError: No module named 'transform_functions' test_trans.py:36: ModuleNotFoundError
tranform_functions is a notebook at the same level as test_trans.py so I am a bit confused as to why I am getting this error...
Full unit test function code which causes the error:
โ05-22-2024 11:01 AM
PS: I have restarted the cluster and ran my run_tests notebook again and now I am getting a different error:
E File "/Workspace/Repos/SBIT/SBIT/test_trans.py", line 36 E from transform_functions import * E ^ E SyntaxError: import * only allowed at module level
I am totally confused now โฆ
โ10-24-2024 06:20 AM
Hi @Kaniz, I am also facing same issue "Unit Testing with PyTest in Databricks - ModuleNotFoundError". Can you please help me too. Because I am not able so see your reply in this thread.
โ10-25-2024 07:54 AM
Hi,
After trying a lot I could able to see some success , see if this is what you all are looking for :
notebook_test.py (this is python code file)
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group