I tried to read a file from S3, but facing the below error:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 53.0 failed 4 times, most recent failure: Lost task 0.3 in stage 53.0 (TID 82, xx.xx.xx.xx, executor 0): com.databricks.sql.io.FileReadException: Error while reading file s3://<mybucket>/<path>/file.csv.

I used:

spark.read.options(delimiter = '|').option("header", False).csv('s3://<mybucket>/<path>/file.csv')