Databricks

FG · ‎04-09-2023

I have a test file (test_transforms.py) which has a series of tests running using Python's unittest package. I can successfully run the tests inside of the file with expected output. But when I try to run this test file from a different notebook (run_unit_tests), it doesn't produce any test results ('Ran 0 tests in 0.000s').

I have tried to keep both files in the same dir but still doesn't work. Here's how the tests are being run:

# print(os.getcwd())

# print(os.listdir())

run_test = unittest.main(argv=[' '], verbosity=2, exit=False)

assert run_test.result.wasSuccessful(), 'Test failed; see logs above'

When run from a different notebook (run_unit_tests)

When run from the test file (test_transforms.py)

NB: I have my notebooks and .py files in a repo. How can I succesfully run testfiles from a different notebook as my intent is to have these tests triggered with Github actions? (I have the workflow set up successfully)

Anonymous · ‎04-11-2023

@Fuad Goloba :

When running tests on Databricks, you need to ensure that the test file is uploaded to the Databricks workspace and that the correct path is specified when importing the test module in the notebook that is running the tests. Here's an example of how you could modify your run_unit_tests notebook to run the tests from test_transforms.py on Databricks:

# Import the required modules
import unittest
import os
 
# Set the path to the directory containing the test file
test_dir = '/path/to/test/files/'
 
# Add the test directory to the Python path
os.sys.path.append(test_dir)
 
# Import the test module
import test_transforms
 
# Run the tests
run_test = unittest.main(argv=[' '], verbosity=2, exit=False)
 
# Check if the tests were successful
assert run_test.result.wasSuccessful(), 'Test failed; see logs above'

Make sure to replace /path/to/test/files/ with the actual path to the directory containing your test file. You can check the path using the dbutils.fs.ls() function. Once you have verified that the tests are running correctly on Databricks, you can set up a GitHub action to trigger the tests as part of your continuous integration process. In the action, you can use the Databricks CLI to upload the test file to the workspace and run the tests on Databricks.

FG · ‎04-11-2023

Thanks for your reply. I have tried the solution proposed now and still doesn't provide any results. Alternatively, I tried to run the tests using unittest's methods by discovering all the test files in the directory and running them (this worked).

Attached is a screenshot of both solutions: