cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Databricks Unity Catalog Shared Mode Cluster Py4J Security Issue

Ian_P
New Contributor II
Hi there, I am getting this error when trying to use Databricks Runtime 13.1, Shared Mode (We need unity catalog), multimode cluster (this works in single user mode, but we need shared mode):
Ian_P_0-1690531566535.png

 

py4j.security.Py4JSecurityException: Method public java.lang.String com.databricks.backend.common.rpc.CommandContext.toJson() is not whitelisted on class class com.databricks.backend.common.rpc.CommandContext
 
I've tried a ton of things online including adding to my spark configs:
spark.databricks.security.py4j.whitelist com.databricks.backend.common.rpc.CommandContext 
spark.driver.extraJavaOptions -Dpy4j.security.allowed.methods=com.databricks.backend.common.rpc.CommandContext
 
but no luck. is there a way to whitelist this that I'm missing? or another work around?
5 REPLIES 5

karthik_p
Esteemed Contributor

@Ian_P can you please add below in spark config settings, looks you missed false, we are not seeing any clear article for reason

https://learn.microsoft.com/en-us/answers/questions/1284871/spark-catalog-whitelisted-for-shared-clu...

spark.databricks.pyspark.enablePy4JSecurity false

Ian_P
New Contributor II

Hi Karthik databricks won't allow me to disable py4j security. when I add that config. it throws an error (saying that I'm not allowed to add that config) in databricks and won't allow me to confirm and restart the cluster 

Yulei
New Contributor III

Hi, I am bumping this post since I am also encountering the same issue with shared cluster with apache sedona class when try to register and getting the following Py4JSecurityException error:


py4j.security.Py4JSecurityException: Method public static void org.apache.sedona.sql.utils.SedonaSQLRegistrator.registerAll(org.apache.spark.sql.SparkSession) is not whitelisted on class class org.apache.sedona.sql.utils.SedonaSQLRegistrator

While we can use the personal compute/single user, but the team still prefer the shared cluster when we are working on the unity catalog. Is there any recommendation on how to add the class to whitelist, or any other aproach?

Ayushi_Suthar
Databricks Employee
Databricks Employee

Hi @Ian_P , Thanks for bringing up your concerns, always happy to help ๐Ÿ˜

Are you getting the py4j.security exeception when trying to schedule a job on the shared cluster?

If yes then could you please share the screenshot of the notebook code that you are scheduling as well? I would like to understand what the code in the Notebook is doing that is encountering the Py4JSecurity. Due to the Table access control and Py4j security, there are some limitations on Shared mode clusters like commands that access the SparkContext. 

Please refer the below doc for the more details about the limitations: 
https://docs.databricks.com/en/data-governance/unity-catalog/compute.html#shared-access-mode-limitat...

Looking forward to hearing from you! 

Ian_P
New Contributor II

@Ayushi_Suthar @Yulei 

After chatting to databricks support, it seems this behaviour is very intentional and there is no work around since the security around Unity Catalog is strict and necessary. We are just using single user cluster.

 

Regards

Ian

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group