Hi there, I am getting this error when trying to use Databricks Runtime 13.1, Shared Mode (We need unity catalog), multimode cluster (this works in single user mode, but we need shared mode):
py4j.security.Py4JSecurityException: Method public java.lang.String com.databricks.backend.common.rpc.CommandContext.toJson() is not whitelisted on class class com.databricks.backend.common.rpc.CommandContext
I've tried a ton of things online including adding to my spark configs:
spark.databricks.security.py4j.whitelist com.databricks.backend.common.rpc.CommandContext
spark.driver.extraJavaOptions -Dpy4j.security.allowed.methods=com.databricks.backend.common.rpc.CommandContext
but no luck. is there a way to whitelist this that I'm missing? or another work around?
