drop duplicates within watermark

aerofish — Wed, 27 Sep 2023 07:16:22 GMT

Recently we are using structured streaming to ingest data. We want to use watermark to drop duplicated event. But We encountered some wired behavior and unexpected exception. Anyone can help me to explain what is the expected behavior and how should I use these method in a right ways?

I have four scenarios:

ingest from json file to delta table: I use withWatermark + dorpDuplicates
behavior: it will drop all duplicates within the watermark and also drop all events (not only duplicated events) older than watermark. Is this expected behavior?
ingest from delta table to delta table: I use withWatermark + dropduplicates
behavior: it will drop all duplicates within the watermark and also drop duplicated event older than watermark
ingest from delta table to delta table; withWatermark + dropDuplicatesWithinWatermark
behavior: I tested with the new introduce method - dropDuplicatesWithinWatermark. Every time It will throw error: java.util.NoSuchElementException: None.get. It's a generic exception. Can anyone explain why I got this error by doing just basic invocation of dropDuplicatesWithinWatermark?
ingest from json file to delta table; withWatermark + dropduplicatewithwatermark
behavior: it will drop duplicates within wartermark, and also drop every event older than watermark. So the behavior is different compare to 3rd scenario(Same method, but from delta table to delta table)

Should I use dropDuplicatesWithinWatermark? it throws exception when doing delta table to delta table ingestion. Is it a bug?

Thanks!

Re: drop duplicates within watermark

Max_Liu — Fri, 29 Sep 2023 02:59:16 GMT

I can confirm we are also getting the same error with the case NO. 3:
> ingest from delta table to delta table; withWatermark + dropDuplicatesWithinWatermark
behavior: I tested with the new introduce method - dropDuplicatesWithinWatermark. Every time It will throw error: java.util.NoSuchElementException: None.get. It's a generic exception.

Re: drop duplicates within watermark

aerofish — Mon, 09 Oct 2023 05:49:31 GMT

Thanks for sharing your experience!

Waiting for more explanation and solutions...

Re: drop duplicates within watermark

aerofish — Wed, 18 Oct 2023 08:16:48 GMT

Any maintainer can help me on this question??

topic Re: drop duplicates within watermark in Get Started Discussions

drop duplicates within watermark

Re: drop duplicates within watermark

Re: drop duplicates within watermark

Re: drop duplicates within watermark