topic Re: dropDuplicates inside foreachBatch in Data Engineering

dropDuplicates inside foreachBatch

MikeGo — Sat, 26 Oct 2024 05:56:22 GMT

Hi,

If I use dropDuplicates inside foreachBatch, the dropDuplicates will become stateless and no state. It just drop duplicates for the current micro batch so I don't have to specify watermark. Is this true?

Thanks

Re: dropDuplicates inside foreachBatch

szymon_dybczak — Sat, 26 Oct 2024 07:22:19 GMT

Hi @MikeGo ,

Yes, you are correct. foreachBatch operates on a single batch at time, so you can safely use drop duplicates without worrying about state management.

Re: dropDuplicates inside foreachBatch

navallyemul — Sun, 27 Oct 2024 17:30:15 GMT

Yes, you're correct! When using dropDuplicates within foreachBatch, it operates only on the current micro-batch, so it removes duplicates in a stateless manner for each batch independently. Since there's no continuous state tracking across batches, you don't need to specify a watermark.

If you need to track duplicates across batches, you'd have to handle state management explicitly or use a different approach.