https://community.databricks.com/t5/data-engineering/autoloader-maxfilespertrigger-not-working-correctly/m-p/3580#M586 <P>Hi @Sanjay Jain​&nbsp;, Since you have provided the trigger as once, the maxFilesPerTrigger will not take effect here. With trigger once, all the files will be read together. You need to change the trigger for this option to come into effect.</P><P>Please refer the document here :</P><P><A href="https://docs.databricks.com/ingestion/auto-loader/options.html#common-auto-loader-options" target="test_blank">https://docs.databricks.com/ingestion/auto-loader/options.html#common-auto-loader-options</A></P> Wed, 07 Jun 2023 19:11:04 GMT Lakshay 2023-06-07T19:11:04Z https://community.databricks.com/t5/data-engineering/autoloader-maxfilespertrigger-not-working-correctly/m-p/3579#M585 <P>Hi,</P><P></P><P>am trying to apply batch size in autoloader and code is as below. But its picking all the changes in one go even if I have put maxFilesPerTrigger&nbsp;as 10. Appreciate any help.</P><P></P><P>(spark.readStream.format("json").schema(streamSchema)</P><P>.option("cloudFiles.backfillInterval", "1 day")</P><P>.option("cloudFiles.fetchParallelism", 100)</P><P>.option("cloudFiles.useNotification","true")</P><P>.option("cloudFiles.includeExistingFiles","true")</P><P>.option("cloudFiles.allowOverwrites",True)</P><P>.option("ignoreMissingFiles",True)</P><P>.option("maxFilesPerTrigger", 10)</P><P>.load(raw_path)</P><P>.writeStream</P><P>.option("checkpointLocation", bronze_checkpoint_path)</P><P>.trigger(once=True)</P><P>.foreachBatch(foreachBatchFunction)</P><P>.start()</P><P>.awaitTermination()</P><P>)</P> Tue, 06 Jun 2023 06:18:17 GMT https://community.databricks.com/t5/data-engineering/autoloader-maxfilespertrigger-not-working-correctly/m-p/3579#M585 sanjay 2023-06-06T06:18:17Z https://community.databricks.com/t5/data-engineering/autoloader-maxfilespertrigger-not-working-correctly/m-p/3580#M586 <P>Hi @Sanjay Jain​&nbsp;, Since you have provided the trigger as once, the maxFilesPerTrigger will not take effect here. With trigger once, all the files will be read together. You need to change the trigger for this option to come into effect.</P><P>Please refer the document here :</P><P><A href="https://docs.databricks.com/ingestion/auto-loader/options.html#common-auto-loader-options" target="test_blank">https://docs.databricks.com/ingestion/auto-loader/options.html#common-auto-loader-options</A></P> Wed, 07 Jun 2023 19:11:04 GMT https://community.databricks.com/t5/data-engineering/autoloader-maxfilespertrigger-not-working-correctly/m-p/3580#M586 Lakshay 2023-06-07T19:11:04Z https://community.databricks.com/t5/data-engineering/autoloader-maxfilespertrigger-not-working-correctly/m-p/3581#M587 <P>Thank you Lakshay</P> Thu, 08 Jun 2023 05:41:01 GMT https://community.databricks.com/t5/data-engineering/autoloader-maxfilespertrigger-not-working-correctly/m-p/3581#M587 sanjay 2023-06-08T05:41:01Z https://community.databricks.com/t5/data-engineering/autoloader-maxfilespertrigger-not-working-correctly/m-p/3582#M588 <P>Hi,</P><P></P><P>This is working for autoloader and I am able to limit batch size. But in next layer, I am still getting full load. Is there any way to limit batch size. Here is my code.</P><P></P><P>(spark.readStream.format("delta")</P><P>.option("cloudFiles.useNotification","true")</P><P>.option("cloudFiles.includeExistingFiles","true")</P><P>.option("cloudFiles.allowOverwrites",True)</P><P>.option("ignoreMissingFiles",True)</P><P>.option("cloudFiles.maxFilesPerTrigger", 100)</P><P>.option("ignoreChanges","true")</P><P>.load(bronze_path)</P><P>.writeStream</P><P>.option("checkpointLocation", silver_checkpoint_path)</P><P>.trigger(processingTime="1 minute")</P><P>.foreachBatch(foreachBatchFunction)</P><P>.start()</P><P>)</P> Fri, 09 Jun 2023 10:20:25 GMT https://community.databricks.com/t5/data-engineering/autoloader-maxfilespertrigger-not-working-correctly/m-p/3582#M588 sanjay 2023-06-09T10:20:25Z