unit testing

[elsirya]

Currently I am creating unit tests for our ETL scripts although the test is not able to recognize sc (SparkContext).

Is there a way to mock SparkContext for a unit test?

Code being tested: df = spark.read.json(sc.parallelize([data]))

Error message received when running PyTest: NameError: name 'sc' is not defined 

[imsabarinath]

You could use Fixtures for SparkSession something like below 

import pytest
from pyspark.sql import SparkSession

@pytest.fixture(scope="session")  # Consider "function" for reuse within a test function
def spark_session():
    spark = SparkSession.builder.appName("test").getOrCreate()
    yield spark
    spark.stop()

[elsirya · Accepted solution]

Was able to get this to work.

What I had to do was instantiate the "sc" variable in the PySpark notebook.

PySpark code:

1. "sc = spark.SparkContext"

Then in the PyTest script we add a "@patch()" statement with the "sc" variable and create a "mock_sc" variable in the function header.

PyTest code:

1. "@patch(<path_to_file>.sc)"
2. "def test_function(mock_sc):"

This solution properly mocked SparkContext within the PyTest script.