SqlContext in DBR 14.3
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-11-2024 10:17 PM
I have a Databricks workspace in GCP and I am using the cluster with the Runtime 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am trying to set the checkpoint directory location using the following command in a notebook:
spark.sparkContext.setCheckpointDir("/FileStore/checkpoint")
The sparkSession used here is the default that is init within the notebook. But I get the following error:
[JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute `sparkContext` is not supported in Spark
Connect as it depends on the JVM. If you need to use this attribute, do not use Spark
Connect when creating your session. Visit https://spark.apache.org/docs/latest/sql-
getting-started.html#starting-point-sparksession for creating regular Spark Session in detail.
It seems in from spark 3.4 we have a spark connect object instead of the the regular sparkSession which does not have the sparkContext attribute. So, as suggested by the error I try to create a spark session using the following and then set the checkpoint directory:
from pyspark.sql import SparkSession
sc = SparkSession \
.builder \
.appName("Python Spark SQL basic example") \
.getOrCreate()
sc.sparkContext.setCheckpointDir("/FileStore/checkpoint")
But I get the same exact error? How do I use the sparkContext attribute in this version of spark and DBR?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-29-2024 02:59 PM
is this error also happening on other DBR versions or only this version shows this message?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-01-2024 07:13 AM
Is this fixed on 14.3 DBR. I am also facing the same when using in Azure Databricks. I am just using df.rdd.getNumPartitions()
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-15-2024 08:50 AM
Same issue with broadcast too. Do you have a solution?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-07-2024 02:24 AM
I have the same issue with sparkContext. I noticed that it only works on single user access mode.
Are there any known fixes/workaround for other access modes?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-22-2024 08:17 AM
Has this been resolved, I am encountering the same issue with df.rdd.getNumPartitions()

