Hi @ChristianRRL
Autoloader ingests your data incrementally regardless of whether you are on directory listing mode or file notification mode. The key difference lies in how it discovers new files. In directory listing mode, Autoloader queries the cloud storage API to list all files in the directory. At scale, this operation becomes more costly because the amount of metadata it needs to list and process grows with every new file added. The cost is tied to the number of I/O operations and the compute resources needed to handle this growing list, which can scale at a greater than linear rate, especially with deeply nested folder structures.
I advise looking at file notification mode, which works much better at scale. It avoids the expensive directory listing by using a pub/sub messaging service (like Azure Event Grid or AWS SQS) to get a notification whenever a new file arrives. This approach is more efficient because the cost scales with the number of new files, not the total number of files in the directory. In my experience, it provided a 10x performance uplift when streaming billions of small binary files, making it a more cost-effective solution.
