sparklyr::spark_read_csv forbidden 403 error

thethirtyfour — Wed, 06 Mar 2024 15:12:52 GMT

Hi,

I am trying to read a csv file into a Spark DataFrame using sparklyr::spark_read_csv. I am receiving a 403 access denied error.

I have stored my AWS credentials as environment variables, and can successfully read the file as an R dataframe using arrow::read_csv_arrow. However, spark_read_csv is failing.

I have confirmed that I am connected to spark, and can read parquet files stored elsewhere.

Any advice?

Thanks,

my_file <- glue::glue("s3://my-bucket/my-folder/my-file-name.csv")

## This works
mydata <- arrow::read_csv_arrow(
file = my_file
)
## This doesn't
mydata <- sparklyr::spark_read_csv(
sc,
name = "mydata"
file = my_file
)

# Error message
Error : java.nio.file.AccessDeniedException

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request

topic sparklyr::spark_read_csv forbidden 403 error in Get Started Discussions

sparklyr::spark_read_csv forbidden 403 error