"spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-06-2023 12:05 AM
Due to some problems, I need to set Py4JSecurity to false, however I get the following message:
"spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"
My cluster is in Shared access mode and a runtime version 12.1 (includes Apache Spark 3.3.1, Scala 2.12) with Unity catalog, Worker type Standard_DS4_v2, and driver type Standard_DS3_v2.
I need it to be a shared cluster that uses unity catalog, so is there any way to get around this?
Thanks!
- Labels:
-
Access Mode
-
Shared Mode
-
Unity Catalog
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-08-2023 10:45 PM
Hi, Could you please test it on a standard standalone cluster/single user cluster and confirm? The error is occurring because enabling the security may break user isolation.
Please let us know if this helps.
Also please tag @Debayan with your next response which will notify me, Thank you!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-08-2023 11:05 PM
Hi @Debayan Mukherjee,
It works in a single user cluster, but I was wondering if it is possible to use it on a shared cluster?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-12-2023 10:43 AM
Hi @Christine Pedersen , It is not possible to use it in a shared cluster as of now. Thank you!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-12-2023 11:48 PM
@Debayan Mukherjee Thank you for the clarification! I will continue with single-user cluster.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-10-2024 04:38 AM
Hello, any update on this? Is this still something not allowed in Databricks? I need to run a function that is not whitelisted and I need that function to run on a shared cluster as I need to read from a table that has a masking in it. So I need either to make possible to run the no whitelisted function in a shared cluster, or I need to be able to read a masked table from a single cluster.
