When Try Returns Success for Invalid S3 Path in Spark: Is This a Bug?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-06-2025 07:33 PM
Try(spark.read.format("parquet").load("s3://abcd/abcd/")) should result in Failure, but when executed in the notebook, it returns Success as shown below. Isn't this a bug?
Try[DataFrame] = Success(...)
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-07-2025 02:57 AM
@allinux The read is a valid way to load data. Why are you expecting a failure? can you please explain?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-07-2025 04:06 AM
When running in spark shell, a Failure is received as shown below. I wonder why it behaves differently.
scala> Try(spark.read.format("parquet").load("/abcd/abcd/"))
res1: scala.util.Try[org.apache.spark.sql.DataFrame] = Failure(org.apache.spark.sql.AnalysisException: [PATH_NOT_FOUND] Path does not exist: file:/abcd/abcd.)