Hi everyone,

I’m currently architecting a high-volume migration for a platform that generates millions of small files daily across thousands of prefixes. As I evaluate the best ingestion strategy using Databricks Auto Loader, I’m running into an architectural trade-off regarding the discovery mode.

While Directory Listing is simpler to set up, we are concerned about the increasing latency as the number of files in the cloud storage grows. On the other hand, File Notification (using AWS SQS/SNS or Azure Event Grid) offers better scalability but introduces additional cloud infrastructure costs and management overhead.

I’d love to hear from those of you managing petabyte-scale Auto Loader streams:

1. At what point (file count or directory depth) did you find Directory Listing became a bottleneck, forcing a switch to File Notification?
2. For those using File Notification, how are you managing the cost and "orphaned" notification events during massive historical backfills?
3. Have you found a "hybrid" approach—using Directory Listing for the initial backfill and then switching to File Notification for the incremental live stream?