- 64800 Views
- 20 replies
- 20 kudos
Resolved! Cannot use RDD and cannot set "spark.databricks.pyspark.enablePy4JSecurity false" for cluster
I have been using "rdd.flatMap(lambda x:x)" for a while to create lists from columns however after I have changed the cluster to a Shared acess mode (to use unity catalog) I get the following error: py4j.security.Py4JSecurityException: Method public ...
- 64800 Views
- 20 replies
- 20 kudos
- 20 kudos
Hi All,I faced the same issue. When i updated the cluster to DBR 15.5 it got resolved, it is not working in DBR 13.3 (my previous cluster version).Well, it is also working in DBR 10.4. I stayed with 15.5 as being the latest. Just try out with differe...
- 20 kudos
- 3404 Views
- 2 replies
- 3 kudos
Resolved! Unity Catalog + Streaming error: method public X is not whitelisted on class DataStreamReader
I want to import data using the autoloader from a S3 bucket into a table which is managed inside a Unity Catalog.Right now, I run the code on an interactive cluster inside a notebook. In the future the code should run in a job cluster. The error I ge...
- 3404 Views
- 2 replies
- 3 kudos