I have been using "rdd.flatMap(lambda x:x)" for a while to create lists from columns however after I have changed the cluster to a Shared acess mode (to use unity catalog) I get the following error:
py4j.security.Py4JSecurityException: Method public org.apache.spark.rdd.RDD org.apache.spark.api.java.JavaRDD.rdd() is not whitelisted on class class org.apache.spark.api.java.JavaRDD
I have tried to solve the error by adding:
"spark.databricks.pyspark.enablePy4JSecurity false"
however I then get the following error:
"spark.databricks.pyspark.enablePy4JSecurity is not allowed when chossing an access mode"
Does anybody know how to use RDD when using a cluster for unity catalouge?
Thank you!