topic Re: SqlContext in DBR 14.3 in Data Engineering

SqlContext in DBR 14.3

tariq — Fri, 12 Apr 2024 05:17:00 GMT

I have a Databricks workspace in GCP and I am using the cluster with the Runtime 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am trying to set the checkpoint directory location using the following command in a notebook:

spark.sparkContext.setCheckpointDir("/FileStore/checkpoint")

The sparkSession used here is the default that is init within the notebook. But I get the following error:

[JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute `sparkContext` is not supported in Spark Connect as it depends on the JVM. If you need to use this attribute, do not use Spark Connect when creating your session. Visit https://spark.apache.org/docs/latest/sql- getting-started.html#starting-point-sparksession for creating regular Spark Session in detail.

It seems in from spark 3.4 we have a spark connect object instead of the the regular sparkSession which does not have the sparkContext attribute. So, as suggested by the error I try to create a spark session using the following and then set the checkpoint directory:

from pyspark.sql import SparkSession sc = SparkSession \ .builder \ .appName("Python Spark SQL basic example") \ .getOrCreate() sc.sparkContext.setCheckpointDir("/FileStore/checkpoint")

But I get the same exact error? How do I use the sparkContext attribute in this version of spark and DBR?

Re: SqlContext in DBR 14.3

jose_gonzalez — Mon, 29 Apr 2024 21:59:42 GMT

is this error also happening on other DBR versions or only this version shows this message?

Re: SqlContext in DBR 14.3

sudipto84goswam — Sat, 01 Jun 2024 14:13:04 GMT

Is this fixed on 14.3 DBR. I am also facing the same when using in Azure Databricks. I am just using df.rdd.getNumPartitions()

Re: SqlContext in DBR 14.3

RamlaSuhra — Mon, 15 Jul 2024 15:50:39 GMT

Same issue with broadcast too. Do you have a solution?

Re: SqlContext in DBR 14.3

biauy — Wed, 07 Aug 2024 09:24:30 GMT

I have the same issue with sparkContext. I noticed that it only works on single user access mode.

Are there any known fixes/workaround for other access modes?

Re: SqlContext in DBR 14.3

Dave1967 — Thu, 22 Aug 2024 15:17:31 GMT

Has this been resolved, I am encountering the same issue with df.rdd.getNumPartitions()

Re: SqlContext in DBR 14.3

Sjors — Mon, 28 Jul 2025 09:31:25 GMT

Has this been resolved? I'm also encountering the same issue with spark.sparkContext.parallelize(). My code is verifiably running on a single user access cluster.