- 64800 Views
- 20 replies
- 20 kudos
Resolved! Cannot use RDD and cannot set "spark.databricks.pyspark.enablePy4JSecurity false" for cluster
I have been using "rdd.flatMap(lambda x:x)" for a while to create lists from columns however after I have changed the cluster to a Shared acess mode (to use unity catalog) I get the following error: py4j.security.Py4JSecurityException: Method public ...
- 64800 Views
- 20 replies
- 20 kudos
- 20 kudos
Hi All,I faced the same issue. When i updated the cluster to DBR 15.5 it got resolved, it is not working in DBR 13.3 (my previous cluster version).Well, it is also working in DBR 10.4. I stayed with 15.5 as being the latest. Just try out with differe...
- 20 kudos
- 14905 Views
- 7 replies
- 4 kudos
Unity Catalog - spark.* functions throwing Py4JSecurityException - org.apache.spark.sql.internal.CatalogImpl.currentCatalog() is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl
I'm looking to migrate onto unity catalog but a number of my data ingestion notebooks throw a securityexception/whitelist errors for numerous spark. functionsIs there some configuration setting I need to enable to whitelist the spark.* methods/functi...
- 14905 Views
- 7 replies
- 4 kudos
- 4 kudos
Hi @Jakub K​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest provid...
- 4 kudos