topic Re: unit testing in Get Started Discussions

unit testing

elsirya — Wed, 17 Jul 2024 13:25:35 GMT

Currently I am creating unit tests for our ETL scripts although the test is not able to recognize sc (SparkContext).

Is there a way to mock SparkContext for a unit test?

Code being tested: df = spark.read.json(sc.parallelize([data]))

Error message received when running PyTest: NameError: name 'sc' is not defined

Re: unit testing

imsabarinath — Wed, 17 Jul 2024 16:27:03 GMT

You could use Fixtures for SparkSession something like below

import pytest from pyspark.sql import SparkSession @pytest.fixture(scope="session") # Consider "function" for reuse within a test function def spark_session(): spark = SparkSession.builder.appName("test").getOrCreate() yield spark spark.stop()

Re: unit testing

elsirya — Thu, 18 Jul 2024 13:06:10 GMT

Was able to get this to work.

What I had to do was instantiate the "sc" variable in the PySpark notebook.

PySpark code:

"sc = spark.SparkContext"

Then in the PyTest script we add a "@patch()" statement with the "sc" variable and create a "mock_sc" variable in the function header.

PyTest code:

"@patch(<path_to_file>.sc)"
"def test_function(mock_sc):"

This solution properly mocked SparkContext within the PyTest script.