topic Re: Unable to read file from S3 in Data Engineering

Unable to read file from S3

LearningDatabri — Tue, 05 Jul 2022 13:36:26 GMT

I tried to read a file from S3, but facing the below error:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 53.0 failed 4 times, most recent failure: Lost task 0.3 in stage 53.0 (TID 82, xx.xx.xx.xx, executor 0): com.databricks.sql.io.FileReadException: Error while reading file s3://<mybucket>/<path>/file.csv.

I used:

spark.read.options(delimiter = '|').option("header", False).csv('s3://<mybucket>/<path>/file.csv')

Re: Unable to read file from S3

Prabakar — Tue, 05 Jul 2022 15:05:30 GMT

Have you validated if the file exists or not?

Is this happening with all files are specific files?

Re: Unable to read file from S3

Sivaprasad1 — Tue, 05 Jul 2022 19:33:36 GMT

Which DBR version are you using? Could you please test it with a different DBR version probably DBR 9.x?

Re: Unable to read file from S3

LearningDatabri — Wed, 06 Jul 2022 10:49:02 GMT

The file exists. Few files are not working and few are working.

Re: Unable to read file from S3

LearningDatabri — Wed, 06 Jul 2022 10:49:05 GMT

I tried with all the available LTS versions.

Re: Unable to read file from S3

Prabakar — Wed, 06 Jul 2022 10:57:02 GMT

I remember I have seen such an issue before. Please check the S3 life cycle management. If the object is migrated to another storage class (mostly archived) then there are possibilities for this error.

Re: Unable to read file from S3

LearningDatabri — Wed, 06 Jul 2022 11:29:28 GMT

Thanks @Prabakar Ammeappin for this information. We have life cycle management set. The files with this error were not used for some time and were Archived. I wonder why files were moved to Glacier in 60 days. Have to revisit the lifecycle rules and change it.

Re: Unable to read file from S3

LearningDatabri — Wed, 06 Jul 2022 11:31:21 GMT

Now it makes sense why I got the error for some files and not for the others.