How to disable spark connect in the databricks compute?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-25-2024 02:47 AM
I want to be able to access the RDD methods of a Dataframe, but it seems that this is not supported in spark connect. I have been trying to disable spark connect in the spark config using,
spark.databricks.service.server.enabled false
but when I check the type of spark it is still `pyspark.sql.connect.session.SparkSession`. How do I disable spark connect?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-25-2024 04:29 AM
If you only want to disable it, set spark.databricks.service.server.enabled to false. However, you might not be able to run your code afterward
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-25-2024 07:10 AM
Maybe you can do something like:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-26-2024 01:14 AM
I want to disable and I tried setting spark.databricks.service.server.enabled to false in the spark config while creating the compute, but when I run the notebook with this compute and I do
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-26-2024 02:55 AM - edited 09-26-2024 02:55 AM
I have found that when the cluster is shared, it automatically uses that type of session, and in that case, I have not been able to disable it. I don't know if this is your situation. I have avoided some problems that I had with the previous clause.
