Hi @thewizard , thanks for your quick reply. Yes, I have to do the deduplication based on my business unique ID. Because the message publisher (neither pub/sub, nor Databricks connector) could cause the duplication due to retry publishing. Therefore I'm relying on dropDuplicatesWithinWatermark to do deduplication, and this method cause message lost...