topic Re: Permission error loading dataframe from azure unity catalog to GCS bucket in Data Engineering

Permission error loading dataframe from azure unity catalog to GCS bucket

kiko_roy — Thu, 30 Nov 2023 11:29:57 GMT

I am creating a data frame by reading a table's data residing in Azure backed unity catalog. I need to write the df or file to GCS bucket. I have configured the spark cluster config using the GCP service account json values.

on running :

df1.write.format("parquet").save("gs://dev-XXXX-analyt-XXXXXXXX") getting error :
Insufficient privileges: User does not have permission SELECT on any file.. what could be the reason or resolution. Need help

Re: Permission error loading dataframe from azure unity catalog to GCS bucket

kiko_roy — Thu, 30 Nov 2023 15:56:29 GMT

Thanks @Retired_mod . The solution did work !!

Re: Permission error loading dataframe from azure unity catalog to GCS bucket

ruloweb — Mon, 15 Apr 2024 14:50:15 GMT

Hi, is there any terraform resource to apply this GRANT or this have to be done always manually?

Re: Permission error loading dataframe from azure unity catalog to GCS bucket

kiko_roy — Tue, 16 Apr 2024 13:56:36 GMT

Hi @ruloweb , apart from granting access manually on any File, one way that resolved the issue for me was to use a cluster with single user.