We have six years of Parquet files organised under yyyy/mm/dd/, with around 100 table files in each daily folder. We want to process these 100 tables using Auto Loader in a Declarative Pipeline with the AUTO CDC API.

If each table reads the same root path with a filename wildcard, will Auto Loader independently discover all files for every table before applying the filter? Could this cause significant discovery overhead?

What folder structure and Auto Loader configuration are recommended as best practice for this scenario?