https://community.databricks.com/t5/data-engineering/how-to-increase-autoloader-speed-while-working-with-s3-aws/m-p/100942#M40485 <P>Hey everyone!&nbsp;<BR />I am trying to switch from a batch processing job to a autoloader (directory listing mode) on my s3 bucket that has millions of files. I am using modifiedAfter to create my initial checkpoint for the autoloader and want to speed up the process. Does anyone have suggestions to tune the workflows to speed up the process?<BR /><BR />I read that autoloader gets 1000 files on every call in s3 and want to know if that bottlenecks the process?</P> Wed, 04 Dec 2024 16:10:06 GMT hk-modi 2024-12-04T16:10:06Z https://community.databricks.com/t5/data-engineering/how-to-increase-autoloader-speed-while-working-with-s3-aws/m-p/100942#M40485 <P>Hey everyone!&nbsp;<BR />I am trying to switch from a batch processing job to a autoloader (directory listing mode) on my s3 bucket that has millions of files. I am using modifiedAfter to create my initial checkpoint for the autoloader and want to speed up the process. Does anyone have suggestions to tune the workflows to speed up the process?<BR /><BR />I read that autoloader gets 1000 files on every call in s3 and want to know if that bottlenecks the process?</P> Wed, 04 Dec 2024 16:10:06 GMT https://community.databricks.com/t5/data-engineering/how-to-increase-autoloader-speed-while-working-with-s3-aws/m-p/100942#M40485 hk-modi 2024-12-04T16:10:06Z https://community.databricks.com/t5/data-engineering/how-to-increase-autoloader-speed-while-working-with-s3-aws/m-p/100945#M40487 <P>What mode are you using? File notification?</P> Wed, 04 Dec 2024 16:41:53 GMT https://community.databricks.com/t5/data-engineering/how-to-increase-autoloader-speed-while-working-with-s3-aws/m-p/100945#M40487 szymon_dybczak 2024-12-04T16:41:53Z