Databricks Unity Catalog Shared Mode Cluster Py4J Security Issue

[Ian_P]

Hi there, I am getting this error when trying to use Databricks Runtime 13.1, Shared Mode (We need unity catalog), multimode cluster (this works in single user mode, but we need shared mode):

py4j.security.Py4JSecurityException: Method public java.lang.String com.databricks.backend.common.rpc.CommandContext.toJson() is not whitelisted on class class com.databricks.backend.common.rpc.CommandContext

I've tried a ton of things online including adding to my spark configs:

spark.databricks.security.py4j.whitelist com.databricks.backend.common.rpc.CommandContext 

spark.driver.extraJavaOptions -Dpy4j.security.allowed.methods=com.databricks.backend.common.rpc.CommandContext

but no luck. is there a way to whitelist this that I'm missing? or another work around?

[karthik_p]

@Ian_P can you please add below in spark config settings, looks you missed false, we are not seeing any clear article for reason

https://learn.microsoft.com/en-us/answers/questions/1284871/spark-catalog-whitelisted-for-shared-clu...

spark.databricks.pyspark.enablePy4JSecurity false

[Ian_P]

Hi Karthik databricks won't allow me to disable py4j security. when I add that config. it throws an error (saying that I'm not allowed to add that config) in databricks and won't allow me to confirm and restart the cluster 

[Yulei]

Hi, I am bumping this post since I am also encountering the same issue with shared cluster with apache sedona class when try to register and getting the following Py4JSecurityException error:

py4j.security.Py4JSecurityException: Method public static void org.apache.sedona.sql.utils.SedonaSQLRegistrator.registerAll(org.apache.spark.sql.SparkSession) is not whitelisted on class class org.apache.sedona.sql.utils.SedonaSQLRegistrator

While we can use the personal compute/single user, but the team still prefer the shared cluster when we are working on the unity catalog. Is there any recommendation on how to add the class to whitelist, or any other aproach?

[Ayushi_Suthar]

Hi @Ian_P , Thanks for bringing up your concerns, always happy to help 😁

Are you getting the py4j.security exeception when trying to schedule a job on the shared cluster?

If yes then could you please share the screenshot of the notebook code that you are scheduling as well? I would like to understand what the code in the Notebook is doing that is encountering the Py4JSecurity. Due to the Table access control and Py4j security, there are some limitations on Shared mode clusters like commands that access the SparkContext. 

Please refer the below doc for the more details about the limitations: 
https://docs.databricks.com/en/data-governance/unity-catalog/compute.html#shared-access-mode-limitat...

Looking forward to hearing from you! 

[Ian_P]

@Ayushi_Suthar @Yulei 

After chatting to databricks support, it seems this behaviour is very intentional and there is no work around since the security around Unity Catalog is strict and necessary. We are just using single user cluster.

Regards

Ian