https://community.databricks.com/t5/data-engineering/reprocessing-the-data-with-auto-loader/m-p/40136#M27136 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/86760">@Eldar_Dragomir</a>&nbsp;</P><P>In order to re-process the data, we have to change the checkpoint directory. This will start processing the files from the beginning. You can use&nbsp;<STRONG>cloudFiles.maxFilesPerTrigger</STRONG>, to limit the number of files getting processed per micro-batch for maintaining the stability of the pipeline.</P> Thu, 17 Aug 2023 04:13:30 GMT Tharun-Kumar 2023-08-17T04:13:30Z https://community.databricks.com/t5/data-engineering/reprocessing-the-data-with-auto-loader/m-p/40127#M27133 <P>Could you please provide me an idea how I can start reprocessing of my data?&nbsp;<BR />Imagine I have a folder in adls gen2 "/test" with binaryFiles. They already processed with current pipeline.&nbsp;<BR />I want to reprocess the data + continue receive new data.&nbsp;<BR />What the settings I have to set for that?<BR />Do I need two "loads" or I can use one with Trigger.AvailableNow with setting of file limitation per batch?</P> Wed, 16 Aug 2023 22:59:38 GMT https://community.databricks.com/t5/data-engineering/reprocessing-the-data-with-auto-loader/m-p/40127#M27133 Eldar_Dragomir 2023-08-16T22:59:38Z https://community.databricks.com/t5/data-engineering/reprocessing-the-data-with-auto-loader/m-p/40136#M27136 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/86760">@Eldar_Dragomir</a>&nbsp;</P><P>In order to re-process the data, we have to change the checkpoint directory. This will start processing the files from the beginning. You can use&nbsp;<STRONG>cloudFiles.maxFilesPerTrigger</STRONG>, to limit the number of files getting processed per micro-batch for maintaining the stability of the pipeline.</P> Thu, 17 Aug 2023 04:13:30 GMT https://community.databricks.com/t5/data-engineering/reprocessing-the-data-with-auto-loader/m-p/40136#M27136 Tharun-Kumar 2023-08-17T04:13:30Z