Friday
Hello everyone,
Around 3PM EST yesterday (2026-11-12), dozens of Lakeflow jobs started failing on both classic and serverless compute with distinct symptoms
1. On classic compute (DBR 15.4), reading a file from S3 would result in a PATH NOT FOUND error
2. On serverless compute (v2 and v5), reading a file from S3 would result in a "'NoneType' has not attribute 'read'" error indicating the SparkSession object was None/null
The issue appears to be resolved on classic compute as of 6PM EST 2026-11-12. The issue is ongoing on serverless compute. This is the simplest bit of code that demonstrates the issue.
from concurrent.futures import ThreadPoolExecutor
from pyspark.sql import SparkSession
def test(foo):
print(SparkSession.getActiveSession())
with ThreadPoolExecutor(max_workers=4) as executor:
future_to_batch = {
executor.submit(test, foo): foo for foo in [โAโ, โBโ, โCโ, โDโ]
}Instead of a SparkSession object appearing in cell output, we see "None". Replacing SparkSession.getActiveSession() with SparkSession.builder.getOrCreate() alleviates the problem.
Is anyone else seeing this?
Friday
Hello @ifishbein, I dug in and here is what I found.
The serverless behavior is explainable, and your workaround is the right one. Serverless runs on Spark Connect, and in the Spark Connect flavor of PySpark SparkSession.getActiveSession() is backed by a plain Python threading.local(). That slot gets filled in the thread that created the session (your notebook's main thread) and nowhere else. A ThreadPoolExecutor worker starts with an empty slot, so getActiveSession() returns None and the first .read on it fails with the exact 'NoneType' object has no attribute 'read' you're seeing. SparkSession.builder.getOrCreate() checks the thread-local first and then falls back to the process-wide default session (your notebook's spark), which is why swapping the call fixed it. Classic compute reaches the session by a different route, which likely explains the different symptom there.
What I can't explain is why it worked before roughly 3PM EST on 11/12. I don't know what changed in the serverless environment that day and I'd rather not guess. Please open a support ticket with a few failing run IDs and timestamps for both the classic and serverless jobs; the classic failures clearing on their own around 6PM is a useful data point for them. Also check the history on status.databricks.com for your region that afternoon.
On the code side, I'd steer away from getActiveSession() inside worker threads entirely, on either compute type. The most portable fix is to pass the session in from the main thread:
from concurrent.futures import ThreadPoolExecutor
def test(foo, spark):
print(spark)
# spark.read.... goes here
with ThreadPoolExecutor(max_workers=4) as executor:
futures = {executor.submit(test, foo, spark): foo for foo in ["A", "B", "C", "D"]}
One SparkSession is safe to share across driver-side threads like this. If you'd rather look the session up inside the thread, SparkSession.active() (PySpark 3.5 and later) returns the active session if there is one and otherwise the default, which is the fallback you actually want. getOrCreate() does the same thing in practice.
One caution if you go looking at pyspark.InheritableThread or inheritable_thread_target as a fix. They were built for pinned thread mode on classic compute, where each Python thread gets its own JVM thread and Spark local properties don't carry over. On Spark Connect they require you to pass the session explicitly, and they propagate the client's thread-locals such as tags, not the active session, so getActiveSession() in the child thread still comes back None. Pass spark in instead.
Takeaway: treat the session as something you hand to a thread, not something a thread goes looking for. That works the same on classic and serverless and survives whatever happened on Wednesday.
References:
getActiveSession(): https://docs.databricks.com/aws/en/pyspark/reference/classes/sparksession/getActiveSessionSparkSession.active(): https://docs.databricks.com/aws/en/pyspark/reference/classes/sparksession/active_active_session and getActiveSession๐ https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/session.pypyspark.InheritableThread: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.InheritableThread.htmlpyspark.inheritable_thread_target: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.inheritable_thread_target.htmlRegards,
Louis
Friday
Hello @ifishbein, I dug in and here is what I found.
The serverless behavior is explainable, and your workaround is the right one. Serverless runs on Spark Connect, and in the Spark Connect flavor of PySpark SparkSession.getActiveSession() is backed by a plain Python threading.local(). That slot gets filled in the thread that created the session (your notebook's main thread) and nowhere else. A ThreadPoolExecutor worker starts with an empty slot, so getActiveSession() returns None and the first .read on it fails with the exact 'NoneType' object has no attribute 'read' you're seeing. SparkSession.builder.getOrCreate() checks the thread-local first and then falls back to the process-wide default session (your notebook's spark), which is why swapping the call fixed it. Classic compute reaches the session by a different route, which likely explains the different symptom there.
What I can't explain is why it worked before roughly 3PM EST on 11/12. I don't know what changed in the serverless environment that day and I'd rather not guess. Please open a support ticket with a few failing run IDs and timestamps for both the classic and serverless jobs; the classic failures clearing on their own around 6PM is a useful data point for them. Also check the history on status.databricks.com for your region that afternoon.
On the code side, I'd steer away from getActiveSession() inside worker threads entirely, on either compute type. The most portable fix is to pass the session in from the main thread:
from concurrent.futures import ThreadPoolExecutor
def test(foo, spark):
print(spark)
# spark.read.... goes here
with ThreadPoolExecutor(max_workers=4) as executor:
futures = {executor.submit(test, foo, spark): foo for foo in ["A", "B", "C", "D"]}
One SparkSession is safe to share across driver-side threads like this. If you'd rather look the session up inside the thread, SparkSession.active() (PySpark 3.5 and later) returns the active session if there is one and otherwise the default, which is the fallback you actually want. getOrCreate() does the same thing in practice.
One caution if you go looking at pyspark.InheritableThread or inheritable_thread_target as a fix. They were built for pinned thread mode on classic compute, where each Python thread gets its own JVM thread and Spark local properties don't carry over. On Spark Connect they require you to pass the session explicitly, and they propagate the client's thread-locals such as tags, not the active session, so getActiveSession() in the child thread still comes back None. Pass spark in instead.
Takeaway: treat the session as something you hand to a thread, not something a thread goes looking for. That works the same on classic and serverless and survives whatever happened on Wednesday.
References:
getActiveSession(): https://docs.databricks.com/aws/en/pyspark/reference/classes/sparksession/getActiveSessionSparkSession.active(): https://docs.databricks.com/aws/en/pyspark/reference/classes/sparksession/active_active_session and getActiveSession๐ https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/session.pypyspark.InheritableThread: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.InheritableThread.htmlpyspark.inheritable_thread_target: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.inheritable_thread_target.htmlRegards,
Louis
Friday
Thanks Louis! I'll reach out to support to see if they can pinpoint a more specific root cause for this becoming a problem yesterday and will refactor our codebase.
Friday
@Louis_Frolio I submitted a ticket to support. Apparently without a paid support contract they won't address the issue even though it appears to be a sudden breaking changing in platform behavior vs user error.