- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-26-2023 04:41 AM
My code:
from great_expectations.datasource import SparkDFDatasource
from pyspark.sql import SparkSession
session_name = 'mk_spark_session'
spark = SparkSession.builder.appName(session_name).getOrCreate()
datasource = SparkDFDatasource(spark)
query = "SELECT * FROM test_large_table limit 10"
df = spark.sql(query)
print(df)
I am running above code in Azure Databricks.
spark version is 3.4.0
pyspark is 3.4.0
py4j is 0.10.9.7
and I am getting below error
py4j.security.Py4JSecurityException: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.
Py4JError: An error occurred while calling None.org.apache.spark.SparkConf. Trace:
py4j.security.Py4JSecurityException: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.
at py4j.security.WhitelistingPy4JSecurityManager.checkConstructor(WhitelistingPy4JSecurityManager.java:451)
at py4j.Gateway.invoke(Gateway.java:256)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)
at py4j.ClientServerConnection.run(ClientServerConnection.java:115)
at java.lang.Thread.run(Thread.java:750)
- Labels:
-
Azure databricks
-
Public
-
Spark
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-14-2023 02:04 PM
Are you using a high concurrency cluster? if you do, please try to run this code on standard cluster
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-14-2023 02:04 PM
Are you using a high concurrency cluster? if you do, please try to run this code on standard cluster
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-22-2023 08:37 AM
@Jose Gonzalez
yes! That was the case and we fixed it by using it on the standard cluster so it's working now.
but thanks anyway!
out of curiosity why it doesn't work on high-concurrency clusters?
Thanks!
Milind

