How to create a SparkSession in jobs run-unit-tests

yhu126 — Mon, 28 Jul 2025 16:33:35 GMT

I’m converting my Python unit tests to run with databricks jobs run-unit-tests.
Each test needs a SparkSession, but every pattern I try

What I tried

1. Create my own local Spark
spark = (SparkSession.builder
.master("local[*]")
.appName("unit-test")
.getOrCreate())
Fails in Databricks with the “shared SparkContext” stack‑trace.

2. Generic builder (no master)
spark = SparkSession.builder.getOrCreate()
Works locally, but in Databricks sometimes raises [MASTER_URL_NOT_SET]
during fixture setup.

3. Try SparkContext first
from pyspark import SparkContext
sc = SparkContext.getOrCreate()
spark = SparkSession.builder.getOrCreate()
I also tried wrapping the logic in a pytest fixture:

pytest.fixture(scope="session")
def spark_session():
spark = SparkSession.getActiveSession()
if spark is None:
spark = (SparkSession.builder
.master("local[*]")
.appName("unit-test")
.getOrCreate())
return spark

…but the builder.master("local[*]") branch is still executed in Databricks,
and the duplicate‑context error appears.

Has anyone solved this cleanly?

Re: How to create a SparkSession in jobs run-unit-tests

szymon_dybczak — Mon, 28 Jul 2025 19:46:54 GMT

Hi @yhu126 ,

Maybe below blog post give you some inspiration:

Writing Unit Tests for PySpark in Databricks: Appr... - Databricks Community - 122398

topic Re: How to create a SparkSession in jobs run-unit-tests in Data Engineering

How to create a SparkSession in jobs run-unit-tests

Re: How to create a SparkSession in jobs run-unit-tests