cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

SqlContext in DBR 14.3

tariq
New Contributor III

I have a Databricks workspace in GCP and I am using the cluster with the Runtime 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am trying to set the checkpoint directory location using the following command in a notebook:

spark.sparkContext.setCheckpointDir("/FileStore/checkpoint")

The sparkSession used here is the default that is init within the notebook. But I get the following error:

[JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute `sparkContext` is not supported in Spark
Connect as it depends on the JVM. If you need to use this attribute, do not use Spark
Connect when creating your session. Visit https://spark.apache.org/docs/latest/sql-
getting-started.html#starting-point-sparksession for creating regular Spark Session in detail.

It seems in from spark 3.4 we have a spark connect object instead of the the regular sparkSession which does not have the sparkContext attribute. So, as suggested by the error I try to create a spark session using the following and then set the checkpoint directory:

from pyspark.sql import SparkSession

sc = SparkSession \
.builder \
.appName("Python Spark SQL basic example") \
.getOrCreate()
sc.sparkContext.setCheckpointDir("/FileStore/checkpoint")

But I get the same exact error? How do I use the sparkContext attribute in this version of spark and DBR?

5 REPLIES 5

jose_gonzalez
Databricks Employee
Databricks Employee

is this error also happening on other DBR versions or only this version shows this message?

sudipto84goswam
New Contributor II

Is this fixed on 14.3 DBR. I am also facing the same when using in Azure Databricks. I am just using df.rdd.getNumPartitions()

sudipto84goswam_0-1717251149018.png

 

RamlaSuhra
New Contributor II

Same issue with broadcast too. Do you have a solution?

biauy
New Contributor II

I have the same issue with sparkContext. I noticed that it only works on single user access mode.

Are there any known fixes/workaround for other access modes?

Dave1967
New Contributor III

Has this been resolved, I am encountering the same issue with df.rdd.getNumPartitions()

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now