sanjay
Valued Contributor II

Thank you Witold, 2 was just an example. I am having thousands of files coming every second and want to limit files per batch otherwise process gets struct if there are too many files in given batch.

I am able to limit the batch size while running single job but not as continuous job. 

(spark.readStream
.option("skipChangeCommits","true")
.option("useNotifications","true")
.option("includeExistingFiles","true")
.option("allowOverwrites",True)
.option("ignoreMissingFiles",True)
.option("maxFilesPerTrigger"2)
.table(table_path)
.writeStream
.option("checkpointLocation", checkpoint_path)
.trigger(processingTime="1 second")
.foreachBatch(foreachBatchFunction)
.start())