Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-05-2022 10:58 PM
Thx @Jose Gonzalez , this makes sense..
What I do not fully understand is the role of the Delta table transaction log in this matter.
E.g. maxFilesPerTrigger is set to 100 files for each micro-batch.
If the Delta transactions of the streaming source looks somewhat like this:E.g. there are 70 files per transaction. Will the micro batch 1 contain files from version 0 and 1, micro batch 2 contain files from version 1 and 2, and so on? So the Delta Table version is not really relevant for the streaming micro badge sizing?