Hi,

am trying to apply batch size in autoloader and code is as below. But its picking all the changes in one go even if I have put maxFilesPerTrigger as 10. Appreciate any help.

(spark.readStream.format("json").schema(streamSchema)

.option("cloudFiles.backfillInterval", "1 day")

.option("cloudFiles.fetchParallelism", 100)

.option("cloudFiles.useNotification","true")

.option("cloudFiles.includeExistingFiles","true")

.option("cloudFiles.allowOverwrites",True)

.option("ignoreMissingFiles",True)

.option("maxFilesPerTrigger", 10)

.load(raw_path)

.writeStream

.option("checkpointLocation", bronze_checkpoint_path)

.trigger(once=True)

.foreachBatch(foreachBatchFunction)

.start()

.awaitTermination()

)