maxFilesPerTrigger not working while loading data from Unity Catalogue table

sanjay
Valued Contributor II

‎08-28-2024 05:08 AM

Hi,

I am using streaming on unity catalogue tables and trying to limit the number of records read in each batch. Here is my code but its not respecting maxFilesPerTrigger, instead reads all available data.

 

(spark.readStream
.option("skipChangeCommits","true")
.option("useNotifications","true")
.option("includeExistingFiles","true")
.option("allowOverwrites",True)
.option("ignoreMissingFiles",True)
.option("maxFilesPerTrigger", 2)
.table(table_path)
.writeStream
.option("checkpointLocation", checkpoint_path)
.trigger(availableNow=True)
.foreachBatch(foreachBatchFunction)
.start()
.awaitTermination())
 
Appreciate any help here.
0 Kudos