topic Re: If 2 users run the same notebook on the same cluster, will they share the same Spark session? in Data Engineering

If 2 users run the same notebook on the same cluster, will they share the same Spark session?

fijoy — Wed, 21 Jun 2023 15:40:14 GMT

Databricks docs here:

https://docs.databricks.com/notebooks/notebook-isolation.html

state that "Every notebook attached to a cluster has a pre-defined variable named spark that represents a SparkSession."

What if 2 users run the same notebook on the same cluster? Will they share the session? Or is the Spark session user-specific also in addition to cluster- & notebook-specific?

Thanks.

Re: If 2 users run the same notebook on the same cluster, will they share the same Spark session?

Anonymous — Thu, 22 Jun 2023 05:49:05 GMT

Hi @Fijoy Vadakkumpadan

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

Re: If 2 users run the same notebook on the same cluster, will they share the same Spark session?

Lakshay — Thu, 22 Jun 2023 19:19:13 GMT

The spark session is isolated at the notebook level and is not isolated at the user level. So, two users accessing the same notebook will be using the same spark session

Re: If 2 users run the same notebook on the same cluster, will they share the same Spark session?

fijoy — Thu, 22 Jun 2023 19:56:52 GMT

Thanks @Lakshay Goel for the response. Follow up question: can the same notebook be attached to 2 different clusters at the same time by 2 different users?

Re: If 2 users run the same notebook on the same cluster, will they share the same Spark session?

Lakshay — Fri, 23 Jun 2023 19:25:02 GMT

No, we cannot attach the same notebook to 2 different clusters.