Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-29-2024 12:46 AM
Thank you Witold, 2 was just an example. I am having thousands of files coming every second and want to limit files per batch otherwise process gets struct if there are too many files in given batch.
I am able to limit the batch size while running single job but not as continuous job.
(spark.readStream
.option("skipChangeCommits","true")
.option("useNotifications","true")
.option("includeExistingFiles","true")
.option("allowOverwrites",True)
.option("ignoreMissingFiles",True)
.option("maxFilesPerTrigger", 2)
.table(table_path)
.writeStream
.option("checkpointLocation", checkpoint_path)
.trigger(processingTime="1 second")
.foreachBatch(foreachBatchFunction)
.start())