Shared access mode and py4j.security.Py4JSecurityException

cszczotka
New Contributor III

‎10-02-2024 06:41 AM

Hi,

We are getting below exception on shared access mode cluster.

py4j.security.Py4JSecurityException: Method public java.lang.String com.databricks.backend.common.rpc.CommandContext.toJson() is not whitelisted on class class com.databricks.backend.common.rpc.CommandContext

I'm trying to find some workaround for this issue  ( other that switch on single user access mode ) like whitelisting this method by setting spark config in my cluster:
spark.driver.extraJavaOption -Dpy4j.security.allowed.methods=com.databricks.backend.common.rpc.CommandContext.toJson spark.executor.extraJavaOptions -Dpy4j.security.allowed.methods=com.databricks.backend.common.rpc.CommandContext.toJson  

Unfortunately it doesn't work. Do you know if syntax is correct or maybe exist other way how to whitelist method for py4j.security.WhitelistingPy4JSecurityManager ? I see that some commercial product running on Databricks has such possibility https://privacera.com/docs/en/whitelist-py4j-security-manager-via-s3-or-dbfs.html

Maybe you know also where can I find code for py4j.security.WhitelistingPy4JSecurityManager ( I don't see anything related with security on https://github.com/py4j/py4j/tree/master/py4j-java ).

Thanks,

 

 

 

 

 

 

0 Kudos

gchandra
Databricks Employee
Databricks Employee

‎10-02-2024 06:48 AM

Low-level APIs and RDDs won't work on UC Shared Compute clusters (as Governance cannot be enforced on those APIs).

https://medium.com/@gchandra/list/py4jsecurityexception-7e720f2743bc



~
0 Kudos

gchandra
Databricks Employee
Databricks Employee

‎10-02-2024 06:50 AM

If you are looking for any specific property using .toJSON() please do share.



~
0 Kudos

cszczotka
New Contributor III

‎10-02-2024 11:52 PM

@gchandra  I'm aware that I can use .safeToJson() but did you see the code of  py4j.security.WhitelistingPy4JSecurityManager ? How it works what can be whitelist what can't be whitelist ? I don't see py4j.security.WhitelistingPy4JSecurityManager on official py4j github. Is this code is closed and owned by Databricks ?

0 Kudos