Access denied error while reading file from S3 to spark

Monika_Bagyal — Tue, 10 Oct 2023 22:19:01 GMT

I'm seeing the access denied error from spark cluster while reading s3 file into notebook.

Running on personal single user compute with LTS 13.3 ML.

configs setup looks like this:

spark.conf.set("spark.hadoop.fs.s3a.access.key", access_id)

spark.conf.set("spark.hadoop.fs.s3a.secret.key", access_key)

spark.conf.set("spark.hadoop.fs.s3a.session.token", session_token)

spark.conf.set("spark.hadoop.fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider")

spark.conf.set("spark.hadoop.fs.s3a.endpoint", "s3.us-east-1.amazonaws.com")

Code block looks like this

file_location = "s3://bucket_name/"

file_type = "parquet"

df = spark.read.format(file_type).load(file_location)

display(df.head())

Error that I'm getting:
java.nio.file.AccessDeniedException: s3://bucket_name/xxx.parquet: getFileStatus ons3://bucket_name/xxx.parquet: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://bucket_name.parquet {} Hadoop 3.3.4, aws-sdk-java/1.12.390 Linux/5.15.0-1045-aws OpenJDK_64-Bit_Server_VM/25.372-b07 java/1.8.0_372 scala/2.12.15 kotlin/1.6.0 vendor/Azul_Systems,_Inc. cfg/retry-mode/legacy com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: RD3ZAB9V0G6C4W7B, Extended Request ID: 7BDXsMzY0O6RwMdKfFLlGuHlw2AkKj0+O2U6vL2UnF1nXzu9sDsVtPVH4qXv5sYzLf8vV65sNdU=, Cloud Provider: AWS, Instance ID: i-06f065a5b0db0e707 credentials-provider: com.amazonaws.auth.AnonymousAWSCredentials credential-header: no-credential-header signature-present: false (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: RD3ZAB9V0G6C4W7B; S3 Extended Request ID: 7BDXsMzY0O6RwMdKfFLlGuHlw2AkKj0+O2U6vL2UnF1nXzu9sDsVtPVH4qXv5sYzLf8vV65sNdU=; Proxy: null), S3 Extended Request ID: 7BDXsMzY0O6RwMdKfFLlGuHlw2AkKj0+O2U6vL2UnF1nXzu9sDsVtPVH4qXv5sYzLf8vV65sNdU=:403 Forbidden

Please help.

topic Access denied error while reading file from S3 to spark in Data Engineering

Access denied error while reading file from S3 to spark