In your case, Spark isn't automatically pruning partitions because:

Missing Partition Discovery: For Spark to perform partition pruning when reading directly from paths (without a metastore table), you need to explicitly tell it about the partition structure.

Solutions
Option 1: Use basePath with Partition Discovery

DataFrame df = sparkSession.read()
.option("mergeSchema", true)
.option("basePath", "s3://some-bucket/some-path/")
.parquet("s3://some-bucket/some-path/region=na/days=1/");

Option 2: Enable Partition Discovery (Recommended)

DataFrame df = sparkSession.read()
.option("mergeSchema", true)
.option("recursiveFileLookup", "false")
.option("partitionOverwriteMode", "dynamic")
.parquet("s3://some-bucket/some-path/")
.filter("region = 'na' AND days = 1");

// Or more explicitly:
DataFrame df = sparkSession.read()
.option("mergeSchema", true)
.option("basePath", "s3://some-bucket/some-path/")
.parquet("s3://some-bucket/some-path/")
.filter("region = 'na' AND days = 1");