My code:

from great_expectations.datasource import SparkDFDatasource

from pyspark.sql import SparkSession

session_name = 'mk_spark_session'

spark = SparkSession.builder.appName(session_name).getOrCreate()

datasource = SparkDFDatasource(spark)

query = "SELECT * FROM test_large_table limit 10"

df = spark.sql(query)

print(df)

I am running above code in Azure Databricks.

spark version is 3.4.0

pyspark is 3.4.0

py4j is 0.10.9.7

and I am getting below error

py4j.security.Py4JSecurityException: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.

Py4JError: An error occurred while calling None.org.apache.spark.SparkConf. Trace:

py4j.security.Py4JSecurityException: Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.

at py4j.security.WhitelistingPy4JSecurityManager.checkConstructor(WhitelistingPy4JSecurityManager.java:451)

at py4j.Gateway.invoke(Gateway.java:256)

at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)

at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)

at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)

at py4j.ClientServerConnection.run(ClientServerConnection.java:115)

at java.lang.Thread.run(Thread.java:750)