Databricks Community

fostermink

Hi, I have a use case where my spark job running on EMR AWS, and it is reading from a s3 path: some-bucket/some-path/region=na/days=1during my read, I pass DataFrame df = sparkSession.read().option("mergeSchema", true).parquet("some-bucket/some-path...

fostermink

Hi, for option 2, why set recursiveFileLookup false will enable Partition Discovery? from what I read, `recursiveFileLookup default value is false`, so I think in my case it is already option2, the only missing part is basePath?

fostermink

some of my configurationspark.sql.hive.convertMetastoreParquetfalsspark.sql.sources.partitionOverwriteModedynamic

Databricks Community

User Stats

User Activity

Spark aws s3 folder partition pruning doesn't work

Re: Spark aws s3 folder partition pruning doesn't work

Re: Spark aws s3 folder partition pruning doesn't work