https://community.databricks.com/t5/data-engineering/autoloader-cloudfiles-maxfilespertrigger-ignored-with-trigger/m-p/112798#M44333 <P>Hi,&nbsp;</P><P>I'm using the <STRONG>Auto Loader</STRONG> feature to read streaming data from Delta Lake files and process them in a batch. The trigger is set to <STRONG>availableNow</STRONG> to include all new data from the checkpoint offset but I limit the amount of delta files for the batch to be 10 using the <STRONG>cloudFiles.maxFilesPerTrigger</STRONG> option. However, the&nbsp; `process_batch` function always reports that it receives the default 1000 files for its batch. Am I misinterpreting the options here?</P><LI-CODE lang="python">from pyspark.sql import DataFrame, SparkSession from pyspark.sql.functions import col def process_batch(df: DataFrame, batch_id: int) -&gt; None: batch_id: int = batch_id num_files: int = df.select("source_file").distinct().count() num_rows_total: int = df.count() print( f"Batch: '{batch_id}' - Processing {num_files:,} delta files with {num_rows_total:,} rows." ) spark_session: SparkSession = SparkSession.getActiveSession() checkpoint_path: str = "/Volumes/checkpoint_path" table_path: str = "/Volumes/table_path" df: DataFrame = ( spark_session.readStream.format(source="delta") .option(key="cloudFiles.format", value="delta") .option(key="cloudFiles.schemaLocation", value=checkpoint_path) .option(key="cloudFiles.maxFilesPerTrigger", value=10) .load(path=table_path) .select("*", col("_metadata.file_path").alias("source_file")) ) df.writeStream.trigger( availableNow=True ).foreachBatch(func=process_batch).start()</LI-CODE><P>&nbsp;</P> Mon, 17 Mar 2025 12:26:23 GMT johschmidt42 2025-03-17T12:26:23Z https://community.databricks.com/t5/data-engineering/autoloader-cloudfiles-maxfilespertrigger-ignored-with-trigger/m-p/112798#M44333 <P>Hi,&nbsp;</P><P>I'm using the <STRONG>Auto Loader</STRONG> feature to read streaming data from Delta Lake files and process them in a batch. The trigger is set to <STRONG>availableNow</STRONG> to include all new data from the checkpoint offset but I limit the amount of delta files for the batch to be 10 using the <STRONG>cloudFiles.maxFilesPerTrigger</STRONG> option. However, the&nbsp; `process_batch` function always reports that it receives the default 1000 files for its batch. Am I misinterpreting the options here?</P><LI-CODE lang="python">from pyspark.sql import DataFrame, SparkSession from pyspark.sql.functions import col def process_batch(df: DataFrame, batch_id: int) -&gt; None: batch_id: int = batch_id num_files: int = df.select("source_file").distinct().count() num_rows_total: int = df.count() print( f"Batch: '{batch_id}' - Processing {num_files:,} delta files with {num_rows_total:,} rows." ) spark_session: SparkSession = SparkSession.getActiveSession() checkpoint_path: str = "/Volumes/checkpoint_path" table_path: str = "/Volumes/table_path" df: DataFrame = ( spark_session.readStream.format(source="delta") .option(key="cloudFiles.format", value="delta") .option(key="cloudFiles.schemaLocation", value=checkpoint_path) .option(key="cloudFiles.maxFilesPerTrigger", value=10) .load(path=table_path) .select("*", col("_metadata.file_path").alias("source_file")) ) df.writeStream.trigger( availableNow=True ).foreachBatch(func=process_batch).start()</LI-CODE><P>&nbsp;</P> Mon, 17 Mar 2025 12:26:23 GMT https://community.databricks.com/t5/data-engineering/autoloader-cloudfiles-maxfilespertrigger-ignored-with-trigger/m-p/112798#M44333 johschmidt42 2025-03-17T12:26:23Z https://community.databricks.com/t5/data-engineering/autoloader-cloudfiles-maxfilespertrigger-ignored-with-trigger/m-p/112846#M44349 <P>It works when changing "cloudFiles.maxFilesPerTrigger" to "maxFilesPerTrigger". But this is unexpected..</P> Mon, 17 Mar 2025 20:55:36 GMT https://community.databricks.com/t5/data-engineering/autoloader-cloudfiles-maxfilespertrigger-ignored-with-trigger/m-p/112846#M44349 johschmidt42 2025-03-17T20:55:36Z https://community.databricks.com/t5/data-engineering/autoloader-cloudfiles-maxfilespertrigger-ignored-with-trigger/m-p/113687#M44609 <P>In doc it is: "<SPAN>cloudFiles.maxFilesPerTrigger" <span class="lia-unicode-emoji" title=":confused_face:">😕</span><BR /><BR /><A href="https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options" target="_blank">https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options</A><BR /></SPAN></P> Wed, 26 Mar 2025 15:40:33 GMT https://community.databricks.com/t5/data-engineering/autoloader-cloudfiles-maxfilespertrigger-ignored-with-trigger/m-p/113687#M44609 p_romm 2025-03-26T15:40:33Z