RobinK
Contributor

‎05-24-2024 12:44 AM

Update:

Removing the following code from all of our notebook fixed the error:

from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.getOrCreate()
 
I have found a line about SparkSessions in the change logs of databrick-connect: https://docs.databricks.com/en/release-notes/dbconnect/index.html#databricks-connect-1432-python
 
But this still doesn't answer the question, why the error occured in an environment, that did not change at all for us (same DBR Version, same cluster, same code).
 
@Retired_mod maybe you could research if there where any internal updates?