Manisha_Jena
Databricks Employee
Databricks Employee

Hi @dbx_deltaSharin,

When querying the individual partitions, the files are being read by using an S3 access point location while it is using the actual S3 name when reading the table as a whole. This information is fetched from the table metadata itself.

It appears, in the source metastore, the table metadata is pointed to the s3 location where as the partitions are defined with the s3 access point location.

Please review the table and partition metadata at the source table. Update the table metadata and point the table location to the s3 access point similar to what's defined for the partitions.

Also, please review the IAM role which was used, Is it defined to allow access with both the s3 name as well as the s3 access point name? Can we add if one is missing? If this is not in the IAM role, the restriction to the actual S3 bucket from outside may be on a higher level (eg: AWS SCP policies).

View solution in original post