SqlContext in DBR 14.3

[tariq]

I have a Databricks workspace in GCP and I am using the cluster with the Runtime 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am trying to set the checkpoint directory location using the following command in a notebook:

spark.sparkContext.setCheckpointDir("/FileStore/checkpoint")

The sparkSession used here is the default that is init within the notebook. But I get the following error:

[JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute `sparkContext` is not supported in Spark
Connect as it depends on the JVM. If you need to use this attribute, do not use Spark
Connect when creating your session. Visit https://spark.apache.org/docs/latest/sql-
getting-started.html#starting-point-sparksession for creating regular Spark Session in detail.

It seems in from spark 3.4 we have a spark connect object instead of the the regular sparkSession which does not have the sparkContext attribute. So, as suggested by the error I try to create a spark session using the following and then set the checkpoint directory:

from pyspark.sql import SparkSession

sc = SparkSession \
.builder \
.appName("Python Spark SQL basic example") \
.getOrCreate()
sc.sparkContext.setCheckpointDir("/FileStore/checkpoint")

But I get the same exact error? How do I use the sparkContext attribute in this version of spark and DBR?

[jose_gonzalez]

is this error also happening on other DBR versions or only this version shows this message?

[sudipto84goswam]

Is this fixed on 14.3 DBR. I am also facing the same when using in Azure Databricks. I am just using df.rdd.getNumPartitions()

[RamlaSuhra]

Same issue with broadcast too. Do you have a solution?

[biauy]

I have the same issue with sparkContext. I noticed that it only works on single user access mode.

Are there any known fixes/workaround for other access modes?

[Dave1967]

Has this been resolved, I am encountering the same issue with df.rdd.getNumPartitions()