Fixed interval micro-batches and AvailableNow Trigger
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-25-2024 10:35 PM - edited 03-25-2024 10:54 PM
What is the fundamental difference between Fixed interval micro-batches and AvailableNow Trigger, given that both can consume in micro-batch based on the desire size of the micro batch ?
Is the fundamental difference the fact that AvailableNow shut down when finished and Fixed interval micro-batches never shut down ?
Indeed as far as i am understanding the documentation, AvailableNow does not mean, one micro-batch of everything available, but depending on the size set up, might mean consuming multiple micro-batch up to what was available when the job was triggered. Am I understanding this correctly ?
The other thing i find confusing in the documentation is the idea that the micro-batch size is set up by a property of type maxBytePerTrigger (depending on the data source). If AvailableNow represent one trigger then that is a problem. So does AvailableNow actually means multiple triggers ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-14-2024 08:13 AM
This `maxBytesPerTrigger` does not work for file source right ? Only source like Kafka Source ?

