Comment
Muhammad_Umer
New Contributor III
‎02-09-2025 10:13 PM
‎02-09-2025 10:13 PM

Hi @MuraliTalluri,

Thank you for such a detailed article.

I am following dropDuplicatesWithinWatermark with the same steps as yours. The only difference is that I am using autoloader and reading CSV files as the source and writing data to a Delta table. My stream is using trigger(once=True). I upload the source file and observe the behavior after triggering the code.

 
While inserting the following record, I noticed that it got dropped due to deduplication, not because of the watermark, even though its event time is less than the watermark threshold (i.e., 10:01 < 10:02).
 

p2.png

 

I inserted another record mimicking the same scenario as the above record, and this time, it was dropped due to the watermark as expected.

Could you please guide me on why there is this inconsistency in behavior?

Thanks