cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Governance
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Christine
by • Contributor
  • 31723 Views
  • 12 replies
  • 15 kudos

Resolved! Cannot use RDD and cannot set "spark.databricks.pyspark.enablePy4JSecurity false" for cluster

I have been using "rdd.flatMap(lambda x:x)" for a while to create lists from columns however after I have changed the cluster to a Shared acess mode (to use unity catalog) I get the following error: py4j.security.Py4JSecurityException: Method public ...

Data Governance
Reply
  • 31723 Views
  • 12 replies
  • 15 kudos
View Replies
Latest Reply
KandyKad
New Contributor II
  • 15 kudos

Faced this issue multiple times.Solution:1. Don't use Shared Cluster or cluster without Unity Catalog enabled for running 'rdd' queries on Databricks.2. Instead create a Personal Cluster (Single User) with basic configuration and with Unity Catalog e...

  • 15 kudos
11 More Replies
jakubk
by • Contributor
  • 7683 Views
  • 7 replies
  • 3 kudos

Unity Catalog - spark.* functions throwing Py4JSecurityException - org.apache.spark.sql.internal.CatalogImpl.currentCatalog() is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl

I'm looking to migrate onto unity catalog but a number of my data ingestion notebooks throw a securityexception/whitelist errors for numerous spark. functionsIs there some configuration setting I need to enable to whitelist the spark.* methods/functi...

Data Governance
Reply
  • 7683 Views
  • 7 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Jakub K​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest provid...

  • 3 kudos
6 More Replies
Labels