cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

External volume over S3 Access point

pmarko1711
New Contributor II

Can anybody confirm if  external volumes pointing to S3 access points work in Databricks on AWS?

  • I have S3 bucket, but can only access it via S3 access point. The bucket is KMS encrypted.
  • I created an IAM role that can list and read the S3 access point (and can also use the KMS key, plus it gives read access to the underlying bucket). I double checked that it can browse the S3 access point. 
  • The IAM role is assumable by Databricks and by itself.
  • I registered a storage credential and defined an external location (using the former)
  • I created an external volume that uses the very same external location, and I have READ VOLUME privilege

With that:

  • I can browse the files (of the S3 access point) using the external location; however
  • When I try to browse files via the external volume, I get "Access to the storage bucket is forbidden by AWS." error.

I would assume that if I can browse the S3 access point via the external location, I would also be able to browse it via the (linked) external volume. What am I doing wrong? Do S3 access points work for external volumes?

 

2 REPLIES 2

gchandra
Databricks Employee
Databricks Employee

Please check the volume permissions.



~

pmarko1711
New Contributor II

This look fine to me. I am the owner of the (external) volume and have READ VOLUME privilege on it. (as for the external location I am also its owner and have READ FILES, BROSE, CREATE EXTERNAL TABLE and CREATE EXTERNAL VOLUME)

One additional info I got, it seems to me that  Databricks launches s3:GetBucketOwnershipControls and s3:GetBucketVersioning actions (which in my case are on the bucket possibly denied). If so, why does it do so from the volume, but not from the external location? And is it necessary?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group