My code:
from great_expectations.datasource import SparkDFDatasource
from pyspark.sql import SparkSession
session_name = 'mk_spark_session'
spark = SparkSession.builder.appName(session_name).getOrCreate()
datasource = SparkDFDatasource(spark)
query = "SELECT * FROM test_large_table limit 10"
df = spark.sql(query)
print(df)
I am running above code in Azure Databricks.
spark version is 3.4.0
pyspark is 3.4.0
py4j is 0.10.9.7
and I am getting below error
py4j.security.Py4JSecurityException: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.
Py4JError: An error occurred while calling None.org.apache.spark.SparkConf. Trace:
py4j.security.Py4JSecurityException: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.
at py4j.security.WhitelistingPy4JSecurityManager.checkConstructor(WhitelistingPy4JSecurityManager.java:451)
at py4j.Gateway.invoke(Gateway.java:256)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)
at py4j.ClientServerConnection.run(ClientServerConnection.java:115)
at java.lang.Thread.run(Thread.java:750)