Is it possible to pass a Spark session to other python files?

tomcorbin — Fri, 29 Dec 2023 10:43:40 GMT

I am setting up pytest for my repo. I have my functions in separate python files and run pytest from one notebook. For each testing file, I have to create a new Spark session as follows:

@pytest.fixture(scope="session")

def spark():

spark = (

SparkSession.builder.master("local[1]")

.appName("test")

.config("spark.executors.cores", "1")

.config("spark.executors.instances", "1")

.config("spark.sql.shuffle.partitions", "1")

.getOrCreate()

)

yield spark

spark.stop()

Is it possible to create one Spark session and pass it to the test files that need it?

Re: Is it possible to pass a Spark session to other python files?

tomcorbin — Sun, 31 Dec 2023 17:47:29 GMT

I was able to do it by placing the Spark session fixture in the conftest.py file in the root directory.

topic Is it possible to pass a Spark session to other python files? in Get Started Discussions

Is it possible to pass a Spark session to other python files?

Re: Is it possible to pass a Spark session to other python files?