Shared access mode and py4j.security.Py4JSecurityException
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-02-2024 06:41 AM
Hi,
We are getting below exception on shared access mode cluster.
py4j.security.Py4JSecurityException: Method public java.lang.String com.databricks.backend.common.rpc.CommandContext.toJson() is not whitelisted on class class com.databricks.backend.common.rpc.CommandContext
I'm trying to find some workaround for this issue ( other that switch on single user access mode ) like whitelisting this method by setting spark config in my cluster:
spark.driver.extraJavaOption -Dpy4j.security.allowed.methods=com.databricks.backend.common.rpc.CommandContext.toJson spark.executor.extraJavaOptions -Dpy4j.security.allowed.methods=com.databricks.backend.common.rpc.CommandContext.toJson
Unfortunately it doesn't work. Do you know if syntax is correct or maybe exist other way how to whitelist method for py4j.security.WhitelistingPy4JSecurityManager ? I see that some commercial product running on Databricks has such possibility https://privacera.com/docs/en/whitelist-py4j-security-manager-via-s3-or-dbfs.html
Maybe you know also where can I find code for py4j.security.WhitelistingPy4JSecurityManager ( I don't see anything related with security on https://github.com/py4j/py4j/tree/master/py4j-java ).
Thanks,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-02-2024 06:48 AM
Low-level APIs and RDDs won't work on UC Shared Compute clusters (as Governance cannot be enforced on those APIs).
https://medium.com/@gchandra/list/py4jsecurityexception-7e720f2743bc
~
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-02-2024 06:50 AM
If you are looking for any specific property using .toJSON() please do share.
~
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-02-2024 11:52 PM
@gchandra I'm aware that I can use .safeToJson() but did you see the code of py4j.security.WhitelistingPy4JSecurityManager ? How it works what can be whitelist what can't be whitelist ? I don't see py4j.security.WhitelistingPy4JSecurityManager on official py4j github. Is this code is closed and owned by Databricks ?

