szymon_dybczak
Esteemed Contributor III

Hi @yit ,

1. They are not quite the same. Trigger batch defines how many new files Auto Loader lists for ingestion per streaming trigger (this is controlled as you correctly pointed out by cloudFiles.maxFilesPerTrigger and cloudFiles.maxBytesPerTrigger)

2. Micro-batch - this is your unit of data that the query executes on. If you use .forEachBatch, Spark gives your function one micro-batch at a time.

So, you can think of it in following way: a "trigger batch" of files produces the input data that Spark will turn into a micro-batch. 

As example, if you set trigger batch to have maxFilesPerTrigger=10 then Spark will list at most new files in that trigger. And that set of files will become an input for one micro-batch.

But keep in mind if fewer than 10 files are available, your micro-batch will be smaller.

As about controlling the size of microbatch - you're just doing that by setting maxFilesPerTrigger and maxBytesPerTrigger. Remeber, these settings will produce "input"  that micro-batch will operate on, so hence it directly influence the size of micro-batch.

Configure Structured Streaming batch size on Azure Databricks - Azure Databricks | Microsoft Learn

View solution in original post