unit testing

elsirya
New Contributor III

‎07-17-2024 06:25 AM

Currently I am creating unit tests for our ETL scripts although the test is not able to recognize sc (SparkContext).
Is there a way to mock SparkContext for a unit test?
 
Code being tested: df = spark.read.json(sc.parallelize([data]))
 
Error message received when running PyTest: NameError: name 'sc' is not defined 
0 Kudos

imsabarinath
New Contributor III

‎07-17-2024 09:27 AM

You could use Fixtures for SparkSession something like below 

import pytest
from pyspark.sql import SparkSession

@pytest.fixture(scope="session")  # Consider "function" for reuse within a test function
def spark_session():
    spark = SparkSession.builder.appName("test").getOrCreate()
    yield spark
    spark.stop()

elsirya
New Contributor III

‎07-18-2024 06:06 AM

Was able to get this to work.

What I had to do was instantiate the "sc" variable in the PySpark notebook.

PySpark code:

  1. "sc = spark.SparkContext"

Then in the PyTest script we add a "@patch()" statement with the "sc" variable and create a "mock_sc" variable in the function header.

PyTest code:

  1. "@patch(<path_to_file>.sc)"
  2. "def test_function(mock_sc):"

This solution properly mocked SparkContext within the PyTest script.

View solution in original post