<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic Re: SparkSession.getActiveSession() is Suddenly Broken in Python Multi-threading in Data Engineering</title>
    <link>https://community.databricks.com/t5/data-engineering/sparksession-getactivesession-is-suddenly-broken-in-python-multi/m-p/176368#M57132</link>
    <description>&lt;P&gt;Thanks Louis! I'll reach out to support to see if they can pinpoint a more specific root cause for this becoming a problem yesterday and will refactor our codebase.&lt;/P&gt;</description>
    <pubDate>Fri, 13 Nov 2026 19:39:26 GMT</pubDate>
    <dc:creator>ifishbein</dc:creator>
    <dc:date>2026-11-13T19:39:26Z</dc:date>
    <item>
      <title>SparkSession.getActiveSession() is Suddenly Broken in Python Multi-threading</title>
      <link>https://community.databricks.com/t5/data-engineering/sparksession-getactivesession-is-suddenly-broken-in-python-multi/m-p/176357#M57127</link>
      <description>&lt;P&gt;Hello everyone,&lt;/P&gt;&lt;P&gt;Around 3PM EST yesterday (2026-11-12), dozens of Lakeflow jobs started failing on both classic and serverless compute with distinct symptoms&lt;/P&gt;&lt;P&gt;1. On classic compute (DBR 15.4), reading a file from S3 would result in a PATH NOT FOUND error&lt;/P&gt;&lt;P&gt;2. On serverless compute (v2 and v5), reading a file from S3 would result in a "'NoneType' has not attribute 'read'" error indicating the SparkSession object was None/null&lt;/P&gt;&lt;P&gt;The issue appears to be resolved on classic compute as of 6PM EST 2026-11-12. The issue is ongoing on serverless compute. This is the simplest bit of code that demonstrates the issue.&lt;/P&gt;&lt;LI-CODE lang="python"&gt;from concurrent.futures import ThreadPoolExecutor
from pyspark.sql import SparkSession

def test(foo):
    print(SparkSession.getActiveSession())

with ThreadPoolExecutor(max_workers=4) as executor:
    future_to_batch = {
        executor.submit(test, foo): foo for foo in [“A”, “B”, “C”, “D”]
    }&lt;/LI-CODE&gt;&lt;P&gt;Instead of a SparkSession object appearing in cell output, we see "None". Replacing SparkSession.getActiveSession() with SparkSession.builder.getOrCreate() alleviates the problem.&lt;/P&gt;&lt;P&gt;Is anyone else seeing this?&lt;/P&gt;</description>
      <pubDate>Fri, 13 Nov 2026 15:45:09 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/sparksession-getactivesession-is-suddenly-broken-in-python-multi/m-p/176357#M57127</guid>
      <dc:creator>ifishbein</dc:creator>
      <dc:date>2026-11-13T15:45:09Z</dc:date>
    </item>
    <item>
      <title>Re: SparkSession.getActiveSession() is Suddenly Broken in Python Multi-threading</title>
      <link>https://community.databricks.com/t5/data-engineering/sparksession-getactivesession-is-suddenly-broken-in-python-multi/m-p/176365#M57130</link>
      <description>&lt;P&gt;Hello &lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/271668"&gt;@ifishbein&lt;/a&gt;, I dug in and here is what I found.&lt;/P&gt;
&lt;P&gt;The serverless behavior is explainable, and your workaround is the right one. Serverless runs on Spark Connect, and in the Spark Connect flavor of PySpark &lt;CODE&gt;SparkSession.getActiveSession()&lt;/CODE&gt; is backed by a plain Python &lt;CODE&gt;threading.local()&lt;/CODE&gt;. That slot gets filled in the thread that created the session (your notebook's main thread) and nowhere else. A &lt;CODE&gt;ThreadPoolExecutor&lt;/CODE&gt; worker starts with an empty slot, so &lt;CODE&gt;getActiveSession()&lt;/CODE&gt; returns &lt;CODE&gt;None&lt;/CODE&gt; and the first &lt;CODE&gt;.read&lt;/CODE&gt; on it fails with the exact &lt;CODE&gt;'NoneType' object has no attribute 'read'&lt;/CODE&gt; you're seeing. &lt;CODE&gt;SparkSession.builder.getOrCreate()&lt;/CODE&gt; checks the thread-local first and then falls back to the process-wide default session (your notebook's &lt;CODE&gt;spark&lt;/CODE&gt;), which is why swapping the call fixed it. Classic compute reaches the session by a different route, which likely explains the different symptom there.&lt;/P&gt;
&lt;P&gt;What I can't explain is why it worked before roughly 3PM EST on 11/12. I don't know what changed in the serverless environment that day and I'd rather not guess. Please open a support ticket with a few failing run IDs and timestamps for both the classic and serverless jobs; the classic failures clearing on their own around 6PM is a useful data point for them. Also check the history on status.databricks.com for your region that afternoon.&lt;/P&gt;
&lt;P&gt;On the code side, I'd steer away from &lt;CODE&gt;getActiveSession()&lt;/CODE&gt; inside worker threads entirely, on either compute type. The most portable fix is to pass the session in from the main thread:&lt;/P&gt;
&lt;PRE&gt;&lt;CODE class="language-python"&gt;from concurrent.futures import ThreadPoolExecutor

def test(foo, spark):
    print(spark)
    # spark.read.... goes here

with ThreadPoolExecutor(max_workers=4) as executor:
    futures = {executor.submit(test, foo, spark): foo for foo in ["A", "B", "C", "D"]}
&lt;/CODE&gt;&lt;/PRE&gt;
&lt;P&gt;One &lt;CODE&gt;SparkSession&lt;/CODE&gt; is safe to share across driver-side threads like this. If you'd rather look the session up inside the thread, &lt;CODE&gt;SparkSession.active()&lt;/CODE&gt; (PySpark 3.5 and later) returns the active session if there is one and otherwise the default, which is the fallback you actually want. &lt;CODE&gt;getOrCreate()&lt;/CODE&gt; does the same thing in practice.&lt;/P&gt;
&lt;P&gt;One caution if you go looking at &lt;CODE&gt;pyspark.InheritableThread&lt;/CODE&gt; or &lt;CODE&gt;inheritable_thread_target&lt;/CODE&gt; as a fix. They were built for pinned thread mode on classic compute, where each Python thread gets its own JVM thread and Spark local properties don't carry over. On Spark Connect they require you to pass the session explicitly, and they propagate the client's thread-locals such as tags, not the active session, so &lt;CODE&gt;getActiveSession()&lt;/CODE&gt; in the child thread still comes back &lt;CODE&gt;None&lt;/CODE&gt;. Pass &lt;CODE&gt;spark&lt;/CODE&gt; in instead.&lt;/P&gt;
&lt;P&gt;Takeaway: treat the session as something you hand to a thread, not something a thread goes looking for. That works the same on classic and serverless and survives whatever happened on Wednesday.&lt;/P&gt;
&lt;P&gt;References:&lt;/P&gt;
&lt;UL&gt;
&lt;LI&gt;&lt;CODE&gt;getActiveSession()&lt;/CODE&gt;: &lt;A href="https://docs.databricks.com/aws/en/pyspark/reference/classes/sparksession/getActiveSession" target="_blank"&gt;https://docs.databricks.com/aws/en/pyspark/reference/classes/sparksession/getActiveSession&lt;/A&gt;&lt;/LI&gt;
&lt;LI&gt;&lt;CODE&gt;SparkSession.active()&lt;/CODE&gt;: &lt;A href="https://docs.databricks.com/aws/en/pyspark/reference/classes/sparksession/active" target="_blank"&gt;https://docs.databricks.com/aws/en/pyspark/reference/classes/sparksession/active&lt;/A&gt;&lt;/LI&gt;
&lt;LI&gt;Spark Connect session source (see &lt;CODE&gt;_active_session&lt;/CODE&gt; and &lt;CODE&gt;getActiveSession&lt;/CODE&gt;&lt;span class="lia-unicode-emoji" title=":disappointed_face:"&gt;😞&lt;/span&gt; &lt;A href="https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/session.py" target="_blank"&gt;https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/session.py&lt;/A&gt;&lt;/LI&gt;
&lt;LI&gt;&lt;CODE&gt;pyspark.InheritableThread&lt;/CODE&gt;: &lt;A href="https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.InheritableThread.html" target="_blank"&gt;https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.InheritableThread.html&lt;/A&gt;&lt;/LI&gt;
&lt;LI&gt;&lt;CODE&gt;pyspark.inheritable_thread_target&lt;/CODE&gt;: &lt;A href="https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.inheritable_thread_target.html" target="_blank"&gt;https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.inheritable_thread_target.html&lt;/A&gt;&lt;/LI&gt;
&lt;LI&gt;Databricks Connect behavior in notebooks and jobs: &lt;A href="https://docs.databricks.com/aws/en/dev-tools/databricks-connect/notebooks" target="_blank"&gt;https://docs.databricks.com/aws/en/dev-tools/databricks-connect/notebooks&lt;/A&gt;&lt;/LI&gt;
&lt;LI&gt;Platform status: &lt;A href="https://status.databricks.com" target="_blank"&gt;https://status.databricks.com&lt;/A&gt;&lt;/LI&gt;
&lt;/UL&gt;
&lt;P&gt;Regards,&lt;BR /&gt;Louis&lt;/P&gt;</description>
      <pubDate>Fri, 13 Nov 2026 18:54:12 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/sparksession-getactivesession-is-suddenly-broken-in-python-multi/m-p/176365#M57130</guid>
      <dc:creator>Louis_Frolio</dc:creator>
      <dc:date>2026-11-13T18:54:12Z</dc:date>
    </item>
    <item>
      <title>Re: SparkSession.getActiveSession() is Suddenly Broken in Python Multi-threading</title>
      <link>https://community.databricks.com/t5/data-engineering/sparksession-getactivesession-is-suddenly-broken-in-python-multi/m-p/176368#M57132</link>
      <description>&lt;P&gt;Thanks Louis! I'll reach out to support to see if they can pinpoint a more specific root cause for this becoming a problem yesterday and will refactor our codebase.&lt;/P&gt;</description>
      <pubDate>Fri, 13 Nov 2026 19:39:26 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/sparksession-getactivesession-is-suddenly-broken-in-python-multi/m-p/176368#M57132</guid>
      <dc:creator>ifishbein</dc:creator>
      <dc:date>2026-11-13T19:39:26Z</dc:date>
    </item>
    <item>
      <title>Re: SparkSession.getActiveSession() is Suddenly Broken in Python Multi-threading</title>
      <link>https://community.databricks.com/t5/data-engineering/sparksession-getactivesession-is-suddenly-broken-in-python-multi/m-p/176377#M57136</link>
      <description>&lt;P&gt;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/34815"&gt;@Louis_Frolio&lt;/a&gt;&amp;nbsp;I submitted a ticket to support. Apparently without a paid support contract they won't address the issue even though it appears to be a sudden breaking changing in platform behavior vs user error.&lt;/P&gt;</description>
      <pubDate>Sat, 14 Nov 2026 02:31:38 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/sparksession-getactivesession-is-suddenly-broken-in-python-multi/m-p/176377#M57136</guid>
      <dc:creator>ifishbein</dc:creator>
      <dc:date>2026-11-14T02:31:38Z</dc:date>
    </item>
  </channel>
</rss>

