spark_partition_id() - User does not have permission SELECT on anonymous function
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2024 09:43 AM
I'm trying to verify the partitions assigned to rows.
I'm running something like this:
from pyspark.sql.functions import spark_partition_id
df = spark.read.table("some.uc.table").limit(10)
df = df.repartition(2)
df = df.withColumn("partitionid", spark_partition_id())
display(df)The results in:
Insufficient privileges: User does not have permission SELECT on anonymous function
This really seems like a bug that needs get fixed.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2024 11:59 AM
Hello @jes,
I have validate your failure internally and found that there is already an internal request to address this behavior.
Are you using a shared access mode cluster? As this behavior does not look to be observed when using single access mode.
For now the workaround would be: GRANT SELECT ON ANONYMOUS FUNCTION TO `user`
I will keep you all posted once fix is released.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-08-2024 02:29 AM
Thanks for looking into it.
Yes, it is a shared cluster we use as a dev cluster for creating our pipelines. I suppose we will try to apply the workaround you shared 👍
