Hi @johschmidt42 ,

This is a great question, but the mystery actually lies in the very first line of your read configuration: spark_session.readStream.format(source="delta")

Because you are using .format("delta") instead of .format("cloudFiles"), you are actually using native Delta Structured Streaming, not Auto Loader!

Here is exactly why you saw that behavior:

• Why cloudFiles was ignored: Spark silently ignores options that don't apply to the chosen format.

• Why maxFilesPerTrigger worked: That is the correct, native option for controlling rate limits in a standard Delta stream.

The good news? You accidentally did it the right way! Since your source data is already in Delta format, using native Delta streaming (.format("delta")) is much more efficient than using Auto Loader (which is meant for raw files like CSV/JSON).

Option 1 : To clean up your code, you can safely remove the cloudFiles options entirely. Here is the idiomatic way to write it:

df: DataFrame = (
    spark_session.readStream
    .format("delta")
    .option("maxFilesPerTrigger", 10)
    .load(table_path)
    .select("*", col("_metadata.file_path").alias("source_file"))
)

df.writeStream \
  .trigger(availableNow=True) \
  .foreachBatch(process_batch) \
  .start()

df: DataFrame = (
    spark_session.readStream
    .format("cloudFiles") # This invokes Auto Loader
    .option("cloudFiles.format", "parquet") # Must be a raw file format
    .option("cloudFiles.schemaLocation", checkpoint_path)
    .option("cloudFiles.maxFilesPerTrigger", 10) # Now this works!
    .load(raw_files_path)
    .select("*", col("_metadata.file_path").alias("source_file"))
)