SparkSession.getActiveSession() is Suddenly Broken in Python Multi-threading

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

Hello everyone,

Around 3PM EST yesterday (2026-11-12), dozens of Lakeflow jobs started failing on both classic and serverless compute with distinct symptoms

1. On classic compute (DBR 15.4), reading a file from S3 would result in a PATH NOT FOUND error

2. On serverless compute (v2 and v5), reading a file from S3 would result in a "'NoneType' has not attribute 'read'" error indicating the SparkSession object was None/null

The issue appears to be resolved on classic compute as of 6PM EST 2026-11-12. The issue is ongoing on serverless compute. This is the simplest bit of code that demonstrates the issue.

from concurrent.futures import ThreadPoolExecutor
from pyspark.sql import SparkSession

def test(foo):
    print(SparkSession.getActiveSession())

with ThreadPoolExecutor(max_workers=4) as executor:
    future_to_batch = {
        executor.submit(test, foo): foo for foo in [“A”, “B”, “C”, “D”]
    }

Instead of a SparkSession object appearing in cell output, we see "None". Replacing SparkSession.getActiveSession() with SparkSession.builder.getOrCreate() alleviates the problem.

Is anyone else seeing this?