https://community.databricks.com/t5/data-engineering/partition-pruning-with-non-delta-files/m-p/72434#M34565 <P>Howdy - first time caller here.</P><P>I'm trying to figure out how/if partition pruning works with non-CSV files. I have files landing in bronze in physical partitions of the form dlk_load_dtm=&lt;load date time&gt;. I'd like to load the partitions that don't yet appear in silver so I'll be querying with a predicate of dlk_load_dtm=&lt;value not in silver&gt;. Files are CSV, JSON, or whatever I get from the source. Partition is recognized by the external table based on bronze. Will DBX prune these partitions so that I'm not scanning the entire folder structure?</P><P>Thank you in advance!</P> Tue, 11 Jun 2024 20:09:57 GMT jsperson 2024-06-11T20:09:57Z https://community.databricks.com/t5/data-engineering/partition-pruning-with-non-delta-files/m-p/72434#M34565 <P>Howdy - first time caller here.</P><P>I'm trying to figure out how/if partition pruning works with non-CSV files. I have files landing in bronze in physical partitions of the form dlk_load_dtm=&lt;load date time&gt;. I'd like to load the partitions that don't yet appear in silver so I'll be querying with a predicate of dlk_load_dtm=&lt;value not in silver&gt;. Files are CSV, JSON, or whatever I get from the source. Partition is recognized by the external table based on bronze. Will DBX prune these partitions so that I'm not scanning the entire folder structure?</P><P>Thank you in advance!</P> Tue, 11 Jun 2024 20:09:57 GMT https://community.databricks.com/t5/data-engineering/partition-pruning-with-non-delta-files/m-p/72434#M34565 jsperson 2024-06-11T20:09:57Z