dropDuplicates inside foreachBatch

MikeGo
Valued Contributor

Hi,

If I use dropDuplicates inside foreachBatch, the dropDuplicates will become stateless and no state. It just drop duplicates for the current micro batch so I don't have to specify watermark. Is this true?

Thanks

szymon_dybczak
Esteemed Contributor III

Hi @MikeGo ,

Yes, you are correct. foreachBatch operates on a single batch at time, so you can safely use drop duplicates without worrying about state management.

View solution in original post

navallyemul
New Contributor III

Yes, you're correct! When using dropDuplicates within foreachBatch, it operates only on the current micro-batch, so it removes duplicates in a stateless manner for each batch independently. Since there's no continuous state tracking across batches, you don't need to specify a watermark.

If you need to track duplicates across batches, you'd have to handle state management explicitly or use a different approach.