Using AWS access points

marcus1 — Tue, 27 Sep 2022 18:05:10 GMT

I am attempting to read data from an AWS access point but setting the Spark property as described here

https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Configuring_S3_AccessPoints_usage_with_S3A

Modifying the Spark properties

fs.s3a.finance-team-access.accesspoint.arn

arn:aws:s3:us-east-1:123456789012:accesspoint/finance-team-access

When I try to fetch data using dbutils.fs.ls or spark.read it says the bucket "finance-team-access" does not exist (I'm using the example access point from the documentation). It should have attempted.

I'm using databricks runtime 10.4 LTS.

Thank you

Re: Using AWS access points

marcus1 — Wed, 28 Sep 2022 20:41:12 GMT

To answer my own question, the spark propertery is not required.

What is required is for you to use the access point alias, not the configured "name" or "arn" as detailed in the Spark documentation.

Read the access point https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points-policies.html documents very carefully and make sure to add the access policy permission set, as well as the role that you have defined in your profile instance.

This should be a better documented feature in the AWS Databricks documentation.

topic Re: Using AWS access points in Warehousing & Analytics

Using AWS access points

Re: Using AWS access points