topic Re: "spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode" in Data Governance

"spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"

Christine — Mon, 06 Mar 2023 08:05:42 GMT

Due to some problems, I need to set Py4JSecurity to false, however I get the following message:

"spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"

My cluster is in Shared access mode and a runtime version 12.1 (includes Apache Spark 3.3.1, Scala 2.12) with Unity catalog, Worker type Standard_DS4_v2, and driver type Standard_DS3_v2.

I need it to be a shared cluster that uses unity catalog, so is there any way to get around this?

Thanks!

Re: "spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"

Christine — Thu, 09 Mar 2023 07:05:05 GMT

Hi @Debayan Mukherjee,

It works in a single user cluster, but I was wondering if it is possible to use it on a shared cluster?

Re: "spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"

Christine — Mon, 13 Mar 2023 06:48:14 GMT

@Debayan Mukherjee Thank you for the clarification! I will continue with single-user cluster.

Re: "spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"

Debayan — Thu, 09 Mar 2023 06:45:13 GMT

Hi, Could you please test it on a standard standalone cluster/single user cluster and confirm? The error is occurring because enabling the security may break user isolation.

Please let us know if this helps.

Also please tag @Debayan with your next response which will notify me, Thank you!

Re: "spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"

Debayan — Sun, 12 Mar 2023 17:43:18 GMT

Hi @Christine Pedersen , It is not possible to use it in a shared cluster as of now. Thank you!

Re: "spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode&qu

rmartinezdezaya — Tue, 10 Dec 2024 12:38:23 GMT

Hello, any update on this? Is this still something not allowed in Databricks? I need to run a function that is not whitelisted and I need that function to run on a shared cluster as I need to read from a table that has a masking in it. So I need either to make possible to run the no whitelisted function in a shared cluster, or I need to be able to read a masked table from a single cluster.