spark_partition_id() - User does not have permission SELECT on anonymous function

[jes]

I'm trying to verify the partitions assigned to rows.

I'm running something like this:

from pyspark.sql.functions  import spark_partition_id

df = spark.read.table("some.uc.table").limit(10)
df = df.repartition(2)
df = df.withColumn("partitionid", spark_partition_id())
display(df)

The results in: 

Insufficient privileges: User does not have permission SELECT on anonymous function

This really seems like a bug that needs get fixed. 

[Alberto_Umana]

Hello @jes,

I have validate your failure internally and found that there is already an internal request to address this behavior. 

Are you using a shared access mode cluster? As this behavior does not look to be observed when using single access mode.

For now the workaround would be: GRANT SELECT ON ANONYMOUS FUNCTION TO `user`

I will keep you all posted once fix is released.

[jes]

Thanks for looking into it.

Yes, it is a shared cluster we use as a dev cluster for creating our pipelines.  I suppose we will try to apply the workaround you shared 👍