โ07-28-2023 02:32 AM
โ07-28-2023 09:07 AM
@Ian_P can you please add below in spark config settings, looks you missed false, we are not seeing any clear article for reason
spark.databricks.pyspark.enablePy4JSecurity false
โ07-31-2023 01:52 AM
Hi Karthik databricks won't allow me to disable py4j security. when I add that config. it throws an error (saying that I'm not allowed to add that config) in databricks and won't allow me to confirm and restart the cluster
โ02-11-2024 05:22 PM
Hi, I am bumping this post since I am also encountering the same issue with shared cluster with apache sedona class when try to register and getting the following Py4JSecurityException error:
py4j.security.Py4JSecurityException: Method public static void org.apache.sedona.sql.utils.SedonaSQLRegistrator.registerAll(org.apache.spark.sql.SparkSession) is not whitelisted on class class org.apache.sedona.sql.utils.SedonaSQLRegistrator
While we can use the personal compute/single user, but the team still prefer the shared cluster when we are working on the unity catalog. Is there any recommendation on how to add the class to whitelist, or any other aproach?
โ02-11-2024 10:48 PM
Hi @Ian_P , Thanks for bringing up your concerns, always happy to help ๐
Are you getting the py4j.security exeception when trying to schedule a job on the shared cluster?
If yes then could you please share the screenshot of the notebook code that you are scheduling as well? I would like to understand what the code in the Notebook is doing that is encountering the Py4JSecurity. Due to the Table access control and Py4j security, there are some limitations on Shared mode clusters like commands that access the SparkContext.
Please refer the below doc for the more details about the limitations:
https://docs.databricks.com/en/data-governance/unity-catalog/compute.html#shared-access-mode-limitat...
Looking forward to hearing from you!
โ02-15-2024 03:28 AM
After chatting to databricks support, it seems this behaviour is very intentional and there is no work around since the security around Unity Catalog is strict and necessary. We are just using single user cluster.
Regards
Ian
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group