Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.

Mikki007 — Fri, 26 May 2023 11:41:10 GMT

My code:

from great_expectations.datasource import SparkDFDatasource

from pyspark.sql import SparkSession

session_name = 'mk_spark_session'

spark = SparkSession.builder.appName(session_name).getOrCreate()

datasource = SparkDFDatasource(spark)

query = "SELECT * FROM test_large_table limit 10"

df = spark.sql(query)

print(df)

I am running above code in Azure Databricks.

spark version is 3.4.0

pyspark is 3.4.0

py4j is 0.10.9.7

and I am getting below error

py4j.security.Py4JSecurityException: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.

Py4JError: An error occurred while calling None.org.apache.spark.SparkConf. Trace:

py4j.security.Py4JSecurityException: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.

at py4j.security.WhitelistingPy4JSecurityManager.checkConstructor(WhitelistingPy4JSecurityManager.java:451)

at py4j.Gateway.invoke(Gateway.java:256)

at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)

at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)

at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)

at py4j.ClientServerConnection.run(ClientServerConnection.java:115)

at java.lang.Thread.run(Thread.java:750)

Re: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.

jose_gonzalez — Wed, 14 Jun 2023 21:04:26 GMT

Are you using a high concurrency cluster? if you do, please try to run this code on standard cluster

Mikki007 — Thu, 22 Jun 2023 15:37:51 GMT

@Jose Gonzalez

yes! That was the case and we fixed it by using it on the standard cluster so it's working now.

but thanks anyway!

out of curiosity why it doesn't work on high-concurrency clusters?

Thanks!

Milind