topic Re: Spark aws s3 folder partition pruning doesn't work in Data Engineering

Spark aws s3 folder partition pruning doesn't work

fostermink — Fri, 09 May 2025 23:49:54 GMT

Hi, I have a use case where my spark job running on EMR AWS, and it is reading from a s3 path: some-bucket/some-path/region=na/days=1

during my read, I pass

DataFrame df = sparkSession.read().option("mergeSchema", true).parquet("some-bucket/some-path/");

and then I apply filters on df where region=na and days=1.
Shouldn't spark do the partition pruning automatically and then only read this path some-bucket/some-path/region=na/days=1 ?

In my case, I see the spark job reading entire some-bucket/some-path. Why this happen?

Re: Spark aws s3 folder partition pruning doesn't work

fostermink — Fri, 09 May 2025 23:51:48 GMT

some of my configuration

spark.sql.hive.convertMetastoreParquet

fals

spark.sql.sources.partitionOverwriteMode

dynamic

Re: Spark aws s3 folder partition pruning doesn't work

lingareddy_Alva — Sat, 10 May 2025 03:23:33 GMT

In your case, Spark isn't automatically pruning partitions because:

Missing Partition Discovery: For Spark to perform partition pruning when reading directly from paths (without a metastore table), you need to explicitly tell it about the partition structure.

Solutions
Option 1: Use basePath with Partition Discovery

DataFrame df = sparkSession.read()
.option("mergeSchema", true)
.option("basePath", "s3://some-bucket/some-path/")
.parquet("s3://some-bucket/some-path/region=na/days=1/");

Option 2: Enable Partition Discovery (Recommended)

DataFrame df = sparkSession.read()
.option("mergeSchema", true)
.option("recursiveFileLookup", "false")
.option("partitionOverwriteMode", "dynamic")
.parquet("s3://some-bucket/some-path/")
.filter("region = 'na' AND days = 1");

// Or more explicitly:
DataFrame df = sparkSession.read()
.option("mergeSchema", true)
.option("basePath", "s3://some-bucket/some-path/")
.parquet("s3://some-bucket/some-path/")
.filter("region = 'na' AND days = 1");

Re: Spark aws s3 folder partition pruning doesn't work

fostermink — Sat, 10 May 2025 03:41:02 GMT

Hi, for option 2, why set recursiveFileLookup false will enable Partition Discovery? from what I read, `recursiveFileLookup default value is false`, so I think in my case it is already option2, the only missing part is basePath?

Re: Spark aws s3 folder partition pruning doesn't work

lingareddy_Alva — Sat, 10 May 2025 15:38:10 GMT

@fostermink

You're correct that recursiveFileLookup defaults to false, so explicitly setting it doesn't actually change the behavior from the default. I should have been more precise in my explanation.
What's really happening is that when you read from a path without specifying partition information, Spark needs to properly identify the directory structure as partitions rather than just subdirectories.

The most important part is indeed the basePath option:

DataFrame df = sparkSession.read()
.option("mergeSchema", true)
.option("basePath", "s3://some-bucket/some-path/")
.parquet("s3://some-bucket/some-path/")
.filter("region = 'na' AND days = 1");

The basePath tells Spark:

-- This is the root directory for the dataset
-- Any directory structure below this that follows the pattern key=value should be interpreted as partitions
-- When filters are applied on these partition columns, use them for partition pruning

Without the basePath option, Spark might not correctly recognize the partition structure, especially if the schema doesn't explicitly define these columns as partitions.
Additionally, to fully enable partition pruning, these configs can help:

spark.sql.parquet.filterPushdown true
spark.sql.optimizer.dynamicPartitionPruning.enabled true

Re: Spark aws s3 folder partition pruning doesn't work

fostermink — Tue, 24 Jun 2025 20:46:48 GMT

Hi, we tried to use basePath, it doesn't work. We are thinking if the root cause is due to we only store the data in s3, but underlying there are not partition metadata, so spark cannot correctly infer the exact path with given filter, and ended up scanning entire folder. Because we saw different behavior when we stored/read a hudi table on s3(hudi table has partition metadata) and it reads hudi fast.

Thanks

Re: Spark aws s3 folder partition pruning doesn't work

lingareddy_Alva — Wed, 25 Jun 2025 04:38:09 GMT

okay, thanks.

Re: Spark aws s3 folder partition pruning doesn't work

Bheemana — Thu, 05 Feb 2026 21:46:55 GMT

I know it is old post, but people who are looking for answers like me below is what worked for me.

Option 1 (it is crude) but it works. We can also add regex '*' to include list of folders region=*. If you need to satisfy multiple conditions add them as comma separated.

DataFrame df = sparkSession.read()
.option("mergeSchema", true)
.option("basePath", "s3://some-bucket/some-path/")
.parquet("s3://some-bucket/some-path/region=na/days=1/",s3://some-bucket/some-path/region=us-east/days=1*");

Re: Spark aws s3 folder partition pruning doesn't work

pradeep_singh — Thu, 05 Feb 2026 21:58:22 GMT

Does df.printSchema show region and days as partition columns at the end? If not, partition discovery isn’t working.
Can you remove mergeSchema or provide an explicit schema? - With mergeSchema, Spark must read the Parquet footers of all files under the base path to merge column definitions before planning the scan. This happens prior to partition pruning, so you’ll see a read/list of the full tree even if the final scan prunes most files.

Re: Spark aws s3 folder partition pruning doesn't work

balajij8 — Fri, 06 Feb 2026 17:13:51 GMT

You can create a table in the catalog and use it for pruning.