Recently we are using structured streaming to ingest data. We want to use watermark to drop duplicated event. But We encountered some wired behavior and unexpected exception. Anyone can help me to explain what is the expected behavior and how should I use these method in a right ways?
I have four scenarios:
- ingest from json file to delta table: I use withWatermark + dorpDuplicates
behavior: it will drop all duplicates within the watermark and also drop all events (not only duplicated events) older than watermark. Is this expected behavior? - ingest from delta table to delta table: I use withWatermark + dropduplicates
behavior: it will drop all duplicates within the watermark and also drop duplicated event older than watermark - ingest from delta table to delta table; withWatermark + dropDuplicatesWithinWatermark
behavior: I tested with the new introduce method - dropDuplicatesWithinWatermark. Every time It will throw error: java.util.NoSuchElementException: None.get. It's a generic exception. Can anyone explain why I got this error by doing just basic invocation of dropDuplicatesWithinWatermark? - ingest from json file to delta table; withWatermark + dropduplicatewithwatermark
behavior: it will drop duplicates within wartermark, and also drop every event older than watermark. So the behavior is different compare to 3rd scenario(Same method, but from delta table to delta table)
Should I use dropDuplicatesWithinWatermark? it throws exception when doing delta table to delta table ingestion. Is it a bug?
Thanks!
