Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.

[Mikki007]

My code:

from great_expectations.datasource import SparkDFDatasource

from pyspark.sql import SparkSession

session_name = 'mk_spark_session'

spark = SparkSession.builder.appName(session_name).getOrCreate()

datasource = SparkDFDatasource(spark)

query = "SELECT * FROM test_large_table limit 10"

df = spark.sql(query)

print(df)

I am running above code in Azure Databricks.

spark version is 3.4.0

pyspark is 3.4.0

py4j is 0.10.9.7

and I am getting below error

py4j.security.Py4JSecurityException: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.

Py4JError: An error occurred while calling None.org.apache.spark.SparkConf. Trace:

py4j.security.Py4JSecurityException: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.

at py4j.security.WhitelistingPy4JSecurityManager.checkConstructor(WhitelistingPy4JSecurityManager.java:451)

at py4j.Gateway.invoke(Gateway.java:256)

at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)

at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)

at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)

at py4j.ClientServerConnection.run(ClientServerConnection.java:115)

at java.lang.Thread.run(Thread.java:750)

[jose_gonzalez · Accepted solution]

Are you using a high concurrency cluster? if you do, please try to run this code on standard cluster

[Mikki007]

@Jose Gonzalez​ 

yes! That was the case and we fixed it by using it on the standard cluster so it's working now.

but thanks anyway!

out of curiosity why it doesn't work on high-concurrency clusters?

Thanks!

Milind