Always use a brand new, clean checkpoint location when starting your stream to skip existing files.

Example checkpoint path: abfss://<container>@<account>.dfs.core.windows.net/<path>/checkpoints/autoloader_run1

Avoid .trigger(availableNow=True) if you want incremental processing.

Prefer default micro-batch triggers which process new files incrementally, respecting includeExistingFiles=false and checkpoint metadata.

Validate and format timestamps correctly in cloudFiles.modifiedAfter.

Use ISO8601 format with timezone info, e.g., 2025-09-09T00:00:00.000Z.

Ensure that the files to be skipped have correct last modified timestamps matching your filter criteria.

Verify source folder contents.

Remove or archive old files from the source directory if possible.

Use directory listing or file notifications properly configured for your cloud storage source.

Ensure stream is configured with these options:

python
df_stream = spark.readStream \
.format("cloudFiles") \
.option("cloudFiles.format", "text") \
.option("cloudFiles.includeExistingFiles", "false") \
.option("cloudFiles.modifiedAfter", "2025-09-09T00:00:00.000Z") \ # proper ISO8601 format
.load(LANDED_PATH)

query = df_stream.writeStream \
.format("delta") \
.option("checkpointLocation", NEW_CHECKPOINT_PATH) \ # brand new checkpoint path
.start(OUTPUT_PATH)
If old files persist in processing, delete and recreate the checkpoint folder or use a new unique checkpoint location each run.

Use Auto Loader metrics and logs to monitor which files get processed to identify unexpected behaviors.