topic Re: spark_partition_id() - User does not have permission SELECT on anonymous function in Get Started Discussions

spark_partition_id() - User does not have permission SELECT on anonymous function

jes — Thu, 07 Nov 2024 17:43:16 GMT

I'm trying to verify the partitions assigned to rows.

I'm running something like this:

from pyspark.sql.functions import spark_partition_id df = spark.read.table("some.uc.table").limit(10) df = df.repartition(2) df = df.withColumn("partitionid", spark_partition_id()) display(df)

The results in:

Insufficient privileges: User does not have permission SELECT on anonymous function

This really seems like a bug that needs get fixed.

Re: spark_partition_id() - User does not have permission SELECT on anonymous function

Alberto_Umana — Thu, 07 Nov 2024 19:59:37 GMT

Hello @jes,

I have validate your failure internally and found that there is already an internal request to address this behavior.

Are you using a shared access mode cluster? As this behavior does not look to be observed when using single access mode.

For now the workaround would be: GRANT SELECT ON ANONYMOUS FUNCTION TO `user`

I will keep you all posted once fix is released.

Re: spark_partition_id() - User does not have permission SELECT on anonymous function

jes — Fri, 08 Nov 2024 10:29:23 GMT

Thanks for looking into it.

Yes, it is a shared cluster we use as a dev cluster for creating our pipelines. I suppose we will try to apply the workaround you shared 👍