- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-28-2026 12:03 PM
For others who run into this issue:
Changing `cloudFiles.maxFilesPerTrigger` to `maxFilesPerTrigger` is not the solution. Check your checkpoint state first.
If a previous Auto Loader run failed or was cancelled after files had already been discovered/planned, the checkpoint can retain that state. On a later run, changing `cloudFiles.maxFilesPerTrigger` may appear to be ignored because the stream is still working through the files that were already discovered under the previous configuration.
For example, if a run used the default/high value for `cloudFiles.maxFilesPerTrigger`, and then failed inside `foreachBatch` or was cancelled, the checkpoint may already contain a planned batch of files. If you later change the configuration to:
```python
.option("cloudFiles.maxFilesPerTrigger", "1")
```
the next run may still process the previously discovered files as one larger batch, making it look like `cloudFiles.maxFilesPerTrigger` is not being respected.
The cleanest fix is to use a new checkpoint.
If using a new checkpoint is not an option because it would cause a large historical reprocess, another workaround is:
1. Move the affected files out of the Auto Loader source path.
2. Rename them so they will be treated as new files later.
3. Run the stream once with the existing checkpoint. It should complete without processing those files.
4. Move the renamed files back into the source path.
5. Run the stream again with the desired setting, for example:
```python
.option("cloudFiles.maxFilesPerTrigger", "1")
```
At that point, Auto Loader should discover the renamed files as new files and respect the configured rate limit.