I'm curious to get thoughts and experience on this. Intuitively, the directory listing mode makes sense to me in order to ensure that only the latest unprocessed files are picked up and processed, but I'm curious about what the cost impact of this would be as more and more files are added? Do the costs scale linearly?

Is there an easy/clear way to understand how the cost scales with the number of files in the directory? I found the following explanation, but I'm still confused as to what kind of costs I may expect with directory listing in ADLS:

How does directory listing mode work?