Unit tests in notebook not working

RabahO — Wed, 21 Feb 2024 15:39:49 GMT

Hello,

I'm trying to setup a notebook for tests or data quality checks. The name is not important.

I basically read a table (the ETL output process - actual data).

Then I read another table and do the calculation in the notebook (expected data)

I'm stuck at the assertEqual(actual_df, expected_df) part. Basically the assert never works no matter the library I'm using.

I tried with Chispa (a pyspark library for testing, very convenient to avoid doing collects and it help showing the exact row where the differences are) but it didn't work, so I tried with unittest module, but same problem.

It's as if the part where the collect happens is skipped and the assert is never triggered. (the collect works if I do it in any other cell)

Here's some code to show you the logic:

# cell 1 expected_data_query = "select ***" expected_data_df = spark.sql(expected_data_query) # cell 2 actual_data_query = "select ***" actual_data_df = spark.sql(actual_data_query) # cell 3 # starts the pyspark job then they all end up in "skipped state" assert_df_equality(actual_accretio_timechange_df, actual_accretio_timechange_df) # cell 4 # same as cell 3 # can't find the code but I inherited unittest module in a class, made # a unit test function and then ran it in the same way as the documentation says: retcode = pytest.main([".", "-v", "-p", "no:cacheprovider"])

Re: Unit tests in notebook not working

feiyun0112 — Thu, 22 Feb 2024 00:56:16 GMT

you can use nutter, Testing framework for Databricks notebooks

microsoft/nutter: Testing framework for Databricks notebooks (github.com)

Re: Unit tests in notebook not working

RabahO — Mon, 26 Feb 2024 08:39:15 GMT

thank you for the nutter. Tried it and it seems to answer my problematic.

topic Re: Unit tests in notebook not working in Data Engineering

Unit tests in notebook not working

Re: Unit tests in notebook not working

Re: Unit tests in notebook not working