topic Re: How to read data from S3 Access Point by pyspark? in Data Engineering

How to read data from S3 Access Point by pyspark?

yutaro_ono1_558 — Sat, 17 Jul 2021 23:07:17 GMT

I want to read data from s3 access point.

I successfully accessed using boto3 client to data through s3 access point.

s3 = boto3.resource('s3')ap = s3.Bucket('arn:aws:s3:[region]:[aws account id]:accesspoint/[S3 Access Point name]')for obj in ap.objects.all():  print(obj.key)  print(obj.get()['Body'].read())

I tried read access through s3 access point by pyspark.

But, I dose not access to s3 access point with error of " java.lang.NullPointerException: null uri host. This can be caused by unencoded / in the password string".

# Can't access to data
# https://[s3-accesspoint-name]-[accountid].s3-accesspoint.[region].amazonaws.com/[file path]
df = spark.read.csv('s3a://arn:aws:s3:[region]:[aws account id]:accesspoint/[S3 access point name]/[data file path]')
df.show()

How to access through the S3 Access Point to data?

S3 Access Point

https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-access-points.html

Re: How to read data from S3 Access Point by pyspark?

Niclas — Fri, 21 Jan 2022 08:58:19 GMT

Did you get to following up on this issue, Kaniz?

Re: How to read data from S3 Access Point by pyspark?

shrestha-rj — Fri, 10 Nov 2023 19:48:56 GMT

I'm reaching out to seek assistance as I navigate an issue. Currently, I'm trying to read JSON files from an S3 Multi-Region Access Point using a Databricks notebook. While reading directly from the S3 bucket presents no challenges, I encounter an "java.nio.file.AccessDeniedException" error when attempting to read from the Multi-Region Access Point. Any guidance or support you can provide would be greatly appreciated.

spark.read.json("s3://<bucket-name>/").display(). --- No issue

spark.read.json("s3://accesspoint/<ap-name>.mrap").display() -- java.nio.file.AccessDeniedException