Hi @batch_bender ,

Yes, pointing Auto Loader to the root path with a filename wildcard for every table will cause significant file discovery overhead, as each stream independently crawls the entire directory tree (yyyy/mm/dd/).

Best Practice Recommendations:

Enable File Notification: Instead of directory listing, configure Auto Loader with .option("cloudFiles.useNotifications", "true"). This uses cloud event queues (SQS/Event Grid) to send new file paths directly to the streams, avoiding directory scanning altogether.

Re-structure Landing Zone (Ideal): If possible, write files by table first (table_name/yyyy/mm/dd/). This reduces discovery cost to $O(1)$ per table stream.

Use Path Globbing Filter: If you must keep directory listing on the current layout, use .option("cloudFiles.globResourcePath", "*/*/*/*_tableName.parquet") to restrict the search scope efficiently.

Here is the official documentation on Auto Loader directory listing vs file notifications:

https://docs.databricks.com/en/ingestion/auto-loader/file-detection-modes.html

If my answer was helpful, please consider marking it as accepted solution!