Hey everyone!
I am trying to switch from a batch processing job to a autoloader (directory listing mode) on my s3 bucket that has millions of files. I am using modifiedAfter to create my initial checkpoint for the autoloader and want to speed up the process. Does anyone have suggestions to tune the workflows to speed up the process?
I read that autoloader gets 1000 files on every call in s3 and want to know if that bottlenecks the process?
