topic Re: External volume over S3 Access point in Data Engineering

External volume over S3 Access point

pmarko1711 — Wed, 02 Oct 2024 16:17:11 GMT

Can anybody confirm if external volumes pointing to S3 access points work in Databricks on AWS?

I have S3 bucket, but can only access it via S3 access point. The bucket is KMS encrypted.
I created an IAM role that can list and read the S3 access point (and can also use the KMS key, plus it gives read access to the underlying bucket). I double checked that it can browse the S3 access point.
The IAM role is assumable by Databricks and by itself.
I registered a storage credential and defined an external location (using the former)
I created an external volume that uses the very same external location, and I have READ VOLUME privilege

With that:

I can browse the files (of the S3 access point) using the external location; however
When I try to browse files via the external volume, I get "Access to the storage bucket is forbidden by AWS." error.

I would assume that if I can browse the S3 access point via the external location, I would also be able to browse it via the (linked) external volume. What am I doing wrong? Do S3 access points work for external volumes?

Re: External volume over S3 Access point

gchandra — Mon, 07 Oct 2024 12:45:21 GMT

Please check the volume permissions.

Re: External volume over S3 Access point

pmarko1711 — Mon, 07 Oct 2024 13:21:01 GMT

This look fine to me. I am the owner of the (external) volume and have READ VOLUME privilege on it. (as for the external location I am also its owner and have READ FILES, BROSE, CREATE EXTERNAL TABLE and CREATE EXTERNAL VOLUME)

One additional info I got, it seems to me that Databricks launches s3:GetBucketOwnershipControls and s3:GetBucketVersioning actions (which in my case are on the bucket possibly denied). If so, why does it do so from the volume, but not from the external location? And is it necessary?