topic Re: Databricks Unity Catalog Shared Mode Cluster Py4J Security Issue in Data Engineering

Databricks Unity Catalog Shared Mode Cluster Py4J Security Issue

Ian_P — Fri, 28 Jul 2023 09:32:42 GMT

Hi there, I am getting this error when trying to use Databricks Runtime 13.1, Shared Mode (We need unity catalog), multimode cluster (this works in single user mode, but we need shared mode):

py4j.security.Py4JSecurityException: Method public java.lang.String com.databricks.backend.common.rpc.CommandContext.toJson() is not whitelisted on class class com.databricks.backend.common.rpc.CommandContext

I've tried a ton of things online including adding to my spark configs:

spark.databricks.security.py4j.whitelist com.databricks.backend.common.rpc.CommandContext

spark.driver.extraJavaOptions -Dpy4j.security.allowed.methods=com.databricks.backend.common.rpc.CommandContext

but no luck. is there a way to whitelist this that I'm missing? or another work around?

Re: Databricks Unity Catalog Shared Mode Cluster Py4J Security Issue

karthik_p — Fri, 28 Jul 2023 16:07:04 GMT

@Ian_P can you please add below in spark config settings, looks you missed false, we are not seeing any clear article for reason

https://learn.microsoft.com/en-us/answers/questions/1284871/spark-catalog-whitelisted-for-shared-clusters-with

spark.databricks.pyspark.enablePy4JSecurity false

Re: Databricks Unity Catalog Shared Mode Cluster Py4J Security Issue

Ian_P — Mon, 31 Jul 2023 08:52:29 GMT

Hi Karthik databricks won't allow me to disable py4j security. when I add that config. it throws an error (saying that I'm not allowed to add that config) in databricks and won't allow me to confirm and restart the cluster

Re: Databricks Unity Catalog Shared Mode Cluster Py4J Security Issue

Yulei — Mon, 12 Feb 2024 01:22:48 GMT

Hi, I am bumping this post since I am also encountering the same issue with shared cluster with apache sedona class when try to register and getting the following Py4JSecurityException error:

py4j.security.Py4JSecurityException: Method public static void org.apache.sedona.sql.utils.SedonaSQLRegistrator.registerAll(org.apache.spark.sql.SparkSession) is not whitelisted on class class org.apache.sedona.sql.utils.SedonaSQLRegistrator

While we can use the personal compute/single user, but the team still prefer the shared cluster when we are working on the unity catalog. Is there any recommendation on how to add the class to whitelist, or any other aproach?

Re: Databricks Unity Catalog Shared Mode Cluster Py4J Security Issue

Ayushi_Suthar — Mon, 12 Feb 2024 06:48:59 GMT

Hi @Ian_P , Thanks for bringing up your concerns, always happy to help 😁

Are you getting the py4j.security exeception when trying to schedule a job on the shared cluster?

If yes then could you please share the screenshot of the notebook code that you are scheduling as well? I would like to understand what the code in the Notebook is doing that is encountering the Py4JSecurity. Due to the Table access control and Py4j security, there are some limitations on Shared mode clusters like commands that access the SparkContext.

Please refer the below doc for the more details about the limitations:
https://docs.databricks.com/en/data-governance/unity-catalog/compute.html#shared-access-mode-limitations

Looking forward to hearing from you!

Re: Databricks Unity Catalog Shared Mode Cluster Py4J Security Issue

Ian_P — Thu, 15 Feb 2024 11:28:11 GMT

@Ayushi_Suthar @Yulei

After chatting to databricks support, it seems this behaviour is very intentional and there is no work around since the security around Unity Catalog is strict and necessary. We are just using single user cluster.

Regards

Ian

Re: Databricks Unity Catalog Shared Mode Cluster Py4J Security Issue

DB_Learner17 — Tue, 11 Mar 2025 09:20:10 GMT

Hi, i too am working to create a job cluster in databricks workflows which should be unity catalog enabled.But, it works only for single-user mode and not shared,while the team where i work needs it as shared one.I too got the same error like as shown below in highlighted section:

Is any work around still not possible to overcome this?

Also, if its not possible , then i want to know whether it would in anyway affect the compute capacity of the cluster when i switch to single-user mode? i dont want to loose the compute capacity because of this change.