Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
state that "Every notebook attached to a cluster has a pre-defined variable named spark that represents a SparkSession."
What if 2 users run the same notebook on the same cluster? Will they share the session? Or is the Spark session user-specific also in addition to cluster- & notebook-specific?
The spark session is isolated at the notebook level and is not isolated at the user level. So, two users accessing the same notebook will be using the same spark session
The spark session is isolated at the notebook level and is not isolated at the user level. So, two users accessing the same notebook will be using the same spark session
Thanks @Lakshay Goel for the response. Follow up question: can the same notebook be attached to 2 different clusters at the same time by 2 different users?