topic Re: How to disable spark connect in the databricks compute? in Administration & Architecture

How to disable spark connect in the databricks compute?

chinmay0924 — Wed, 25 Sep 2024 09:47:21 GMT

I want to be able to access the RDD methods of a Dataframe, but it seems that this is not supported in spark connect. I have been trying to disable spark connect in the spark config using,

spark.databricks.service.server.enabled false

but when I check the type of spark it is still `pyspark.sql.connect.session.SparkSession`. How do I disable spark connect?

Re: How to disable spark connect in the databricks compute?

Bareaj — Wed, 25 Sep 2024 11:29:48 GMT

If you only want to disable it, set spark.databricks.service.server.enabled to false. However, you might not be able to run your code afterward

Re: How to disable spark connect in the databricks compute?

Bareaj — Wed, 25 Sep 2024 14:10:36 GMT

Maybe you can do something like:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

if spark.__class__.__module__ == "pyspark.sql.connect.session":

code when it is enable

else:

code when not

Re: How to disable spark connect in the databricks compute?

chinmay0924 — Thu, 26 Sep 2024 08:14:54 GMT

I want to disable and I tried setting spark.databricks.service.server.enabled to false in the spark config while creating the compute, but when I run the notebook with this compute and I do

type(spark), I get pyspark.sql.connect.session.SparkSession.

Re: How to disable spark connect in the databricks compute?

Bareaj — Thu, 26 Sep 2024 09:55:35 GMT

I have found that when the cluster is shared, it automatically uses that type of session, and in that case, I have not been able to disable it. I don't know if this is your situation. I have avoided some problems that I had with the previous clause.