"spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"

[Christine]

Due to some problems, I need to set Py4JSecurity to false, however I get the following message:

"spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"

My cluster is in Shared access mode and a runtime version 12.1 (includes Apache Spark 3.3.1, Scala 2.12) with Unity catalog, Worker type Standard_DS4_v2, and driver type Standard_DS3_v2.

I need it to be a shared cluster that uses unity catalog, so is there any way to get around this?

Thanks!

[Debayan]

Hi, Could you please test it on a standard standalone cluster/single user cluster and confirm? The error is occurring because enabling the security may break user isolation.

Please let us know if this helps. 

Also please tag @Debayan​ with your next response which will notify me, Thank you!

[Christine]

Hi @Debayan Mukherjee​,

It works in a single user cluster, but I was wondering if it is possible to use it on a shared cluster?

[Debayan]

Hi @Christine Pedersen​ , It is not possible to use it in a shared cluster as of now. Thank you!

[Christine]

@Debayan Mukherjee​ Thank you for the clarification! I will continue with single-user cluster.

[rmartinezdezaya]

Hello, any update on this? Is this still something not allowed in Databricks? I need to run a function that is not whitelisted and I need that function to run on a shared cluster as I need to read from a table that has a masking in it. So I need either to make possible to run the no whitelisted function in a shared cluster, or I need to be able to read a masked table from a single cluster.