Spark aws s3 folder partition pruning doesn't work
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-09-2025 04:49 PM
Hi, I have a use case where my spark job running on EMR AWS, and it is reading from a s3 path: some-bucket/some-path/region=na/days=1
during my read, I pass
DataFrame df = sparkSession.read().option("mergeSchema", true).parquet("some-bucket/some-path/");
and then I apply filters on df where region=na and days=1.
Shouldn't spark do the partition pruning automatically and then only read this path some-bucket/some-path/region=na/days=1 ?
In my case, I see the spark job reading entire some-bucket/some-path. Why this happen?
