topic Re: When Try Returns Success for Invalid S3 Path in Spark: Is This a Bug? in Data Engineering

When Try Returns Success for Invalid S3 Path in Spark: Is This a Bug?

allinux — Fri, 07 Feb 2025 03:33:46 GMT

Try(spark.read.format("parquet").load("s3://abcd/abcd/")) should result in Failure, but when executed in the notebook, it returns Success as shown below. Isn't this a bug?

Try[DataFrame] = Success(...)

Re: When Try Returns Success for Invalid S3 Path in Spark: Is This a Bug?

MuthuLakshmi — Fri, 07 Feb 2025 10:57:48 GMT

@allinux The read is a valid way to load data. Why are you expecting a failure? can you please explain?

Re: When Try Returns Success for Invalid S3 Path in Spark: Is This a Bug?

allinux — Fri, 07 Feb 2025 12:06:04 GMT

When running in spark shell, a Failure is received as shown below. I wonder why it behaves differently.

scala> Try(spark.read.format("parquet").load("/abcd/abcd/")) res1: scala.util.Try[org.apache.spark.sql.DataFrame] = Failure(org.apache.spark.sql.AnalysisException: [PATH_NOT_FOUND] Path does not exist: file:/abcd/abcd.)