Unable to read file from S3

[LearningDatabri]

I tried to read a file from S3, but facing the below error:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 53.0 failed 4 times, most recent failure: Lost task 0.3 in stage 53.0 (TID 82, xx.xx.xx.xx, executor 0): com.databricks.sql.io.FileReadException: Error while reading file s3://<mybucket>/<path>/file.csv.

I used:

spark.read.options(delimiter = '|').option("header", False).csv('s3://<mybucket>/<path>/file.csv')

[Prabakar]

Have you validated if the file exists or not?

Is this happening with all files are specific files?

[LearningDatabri]

The file exists. Few files are not working and few are working.

[Prabakar · Accepted solution]

I remember I have seen such an issue before. Please check the S3 life cycle management. If the object is migrated to another storage class (mostly archived) then there are possibilities for this error.

[LearningDatabri]

Thanks @Prabakar Ammeappin​  for this information. We have life cycle management set. The files with this error were not used for some time and were Archived. I wonder why files were moved to Glacier in 60 days. Have to revisit the lifecycle rules and change it.

[LearningDatabri]

Now it makes sense why I got the error for some files and not for the others.

[Sivaprasad1]

Which DBR version are you using? Could you please test it with a different DBR version probably DBR 9.x?

[LearningDatabri]

I tried with all the available LTS versions.