topic Re: Shared access mode and py4j.security.Py4JSecurityException in Data Engineering

Shared access mode and py4j.security.Py4JSecurityException

cszczotka — Wed, 02 Oct 2024 13:41:41 GMT

Hi,

We are getting below exception on shared access mode cluster.

py4j.security.Py4JSecurityException: Method public java.lang.String com.databricks.backend.common.rpc.CommandContext.toJson() is not whitelisted on class class com.databricks.backend.common.rpc.CommandContext

I'm trying to find some workaround for this issue ( other that switch on single user access mode ) like whitelisting this method by setting spark config in my cluster:
spark.driver.extraJavaOption -Dpy4j.security.allowed.methods=com.databricks.backend.common.rpc.CommandContext.toJson spark.executor.extraJavaOptions -Dpy4j.security.allowed.methods=com.databricks.backend.common.rpc.CommandContext.toJson

Unfortunately it doesn't work. Do you know if syntax is correct or maybe exist other way how to whitelist method for py4j.security.WhitelistingPy4JSecurityManager ? I see that some commercial product running on Databricks has such possibility https://privacera.com/docs/en/whitelist-py4j-security-manager-via-s3-or-dbfs.html

Maybe you know also where can I find code for py4j.security.WhitelistingPy4JSecurityManager ( I don't see anything related with security on https://github.com/py4j/py4j/tree/master/py4j-java ).

Thanks,

Re: Shared access mode and py4j.security.Py4JSecurityException

gchandra — Wed, 02 Oct 2024 13:48:27 GMT

Low-level APIs and RDDs won't work on UC Shared Compute clusters (as Governance cannot be enforced on those APIs).

https://medium.com/@gchandra/list/py4jsecurityexception-7e720f2743bc

Re: Shared access mode and py4j.security.Py4JSecurityException

gchandra — Wed, 02 Oct 2024 13:50:39 GMT

If you are looking for any specific property using .toJSON() please do share.

Re: Shared access mode and py4j.security.Py4JSecurityException

cszczotka — Thu, 03 Oct 2024 06:52:34 GMT

@gchandra I'm aware that I can use .safeToJson() but did you see the code of py4j.security.WhitelistingPy4JSecurityManager ? How it works what can be whitelist what can't be whitelist ? I don't see py4j.security.WhitelistingPy4JSecurityManager on official py4j github. Is this code is closed and owned by Databricks ?