dropDuplicates inside foreachBatch

[Brad]

Hi,

If I use dropDuplicates inside foreachBatch, the dropDuplicates will become stateless and no state. It just drop duplicates for the current micro batch so I don't have to specify watermark. Is this true?

Thanks

[szymon_dybczak · Accepted solution]

Hi @Brad ,

Yes, you are correct. foreachBatch operates on a single batch at time, so you can safely use drop duplicates without worrying about state management.

[navallyemul]

Yes, you're correct! When using dropDuplicates within foreachBatch, it operates only on the current micro-batch, so it removes duplicates in a stateless manner for each batch independently. Since there's no continuous state tracking across batches, you don't need to specify a watermark.

If you need to track duplicates across batches, you'd have to handle state management explicitly or use a different approach.