topic Re: Access denied error to S3 bucket while running Kinesis spark streaming. in Machine Learning

Access denied error to S3 bucket while running Kinesis spark streaming.

Avi711 — Thu, 10 Nov 2022 07:47:40 GMT

I get this below error while trying to simulate kinesis streams as mentioned in Databricks

documentation at https://docs.databricks.com/getting-started/streaming.html

Error:

java.nio.file.AccessDeniedException:

Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID:

com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied; request: PUT https://..

Does anyone faced this issue. Any pointers will be very helpful.

Re: Access denied error to S3 bucket while running Kinesis spark streaming.

Pat — Thu, 10 Nov 2022 08:08:22 GMT

Hi @Aviraj Madkaikar ,

Access Denied; request: PUT... . It looks like you don't have permission to write to the location.

Do you use .checkpoint or write data somewhere?

Re: Access denied error to S3 bucket while running Kinesis spark streaming.

Avi711 — Sun, 13 Nov 2022 18:50:49 GMT

It was an access issue to S3 bucket, resolved by providing appropriate access. Thanks

Re: Access denied error to S3 bucket while running Kinesis spark streaming.

jsuarezg — Tue, 09 Jan 2024 21:21:16 GMT

If you do

spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3a.access.key", AWS_ACCESS_KEY_ID) + secret with any other secret that has less access than your default one this sometimes happens, so running those commands but with your normal secret will fix the issue