Fixed interval micro-batches and AvailableNow Trigger

Maatari — Tue, 26 Mar 2024 05:54:59 GMT

What is the fundamental difference between Fixed interval micro-batches and AvailableNow Trigger, given that both can consume in micro-batch based on the desire size of the micro batch ?

Is the fundamental difference the fact that AvailableNow shut down when finished and Fixed interval micro-batches never shut down ?

Indeed as far as i am understanding the documentation, AvailableNow does not mean, one micro-batch of everything available, but depending on the size set up, might mean consuming multiple micro-batch up to what was available when the job was triggered. Am I understanding this correctly ?

The other thing i find confusing in the documentation is the idea that the micro-batch size is set up by a property of type maxBytePerTrigger (depending on the data source). If AvailableNow represent one trigger then that is a problem. So does AvailableNow actually means multiple triggers ?

Re: Fixed interval micro-batches and AvailableNow Trigger

Maatari — Tue, 14 May 2024 15:13:55 GMT

This `maxBytesPerTrigger` does not work for file source right ? Only source like Kafka Source ?

topic Re: Fixed interval micro-batches and AvailableNow Trigger in Data Engineering

Fixed interval micro-batches and AvailableNow Trigger

Re: Fixed interval micro-batches and AvailableNow Trigger