Permission error loading dataframe from azure unity catalog to GCS bucket

[kiko_roy]

I am creating a data frame by reading a table's data residing in Azure backed unity catalog. I need to write the df or file to GCS bucket. I have configured the spark cluster config using the GCP service account json values.

on running : 

df1.write.format("parquet").save("gs://dev-XXXX-analyt-XXXXXXXX") getting error :
Insufficient privileges: User does not have permission SELECT on any file.. what could be the reason or resolution. Need help

[Kaniz_Fatma · Accepted solution]

Hi @kiko_roy, The error message you’re encountering, “Insufficient privileges: User does not have permission SELECT on any file,” indicates that your user account lacks the necessary permissions to read files.

Let’s address this issue:

Cause:

• Table access control is enabled on your Databricks cluster, and you are not an admin.
• The Databricks SQL query analyzer enforces access control policies at runtime. When table access control is enabled, users must have specific permissions to access tables.

Solution:

• Admins can bypass table access control, but regular users need explicit permissions.
• An admin must grant SELECT permission on files so that you can create a table.
• Run the following command in a notebook (as an admin):%sql GRANT SELECT ON ANY FILE TO <user@domain-name> Replace <user@domain-name> with your actual user identifier.

Warning:

• Users granted access to ANY FILE can bypass restrictions on the catalog, schemas, tables, and views by reading directly from the filesystem.
• Review the Data object privileges documentation for more information.

Remember to execute the command as an admin, and ensure that the necessary permissions are granted. This should resolve the issue you’re facing.

If you encounter any further difficulties, feel free to ask for additional assistance! 🚀

[kiko_roy]

Thanks @Kaniz_Fatma . The solution did work !!

[Kaniz_Fatma]

I want to express my gratitude for your effort in selecting the most suitable solution. It's great to hear that your query has been successfully resolved. Thank you for your contribution.




 

[ruloweb]

Hi, is there any terraform resource to apply this GRANT or this have to be done always manually?

[kiko_roy]

Hi @ruloweb , apart from granting access manually on any File, one way that resolved the issue for me was to use a cluster with single user.