Streaming Query

[nileshtiwaari]

How to remove duplicates in streaming query on the basis of some id?

[daniel_sahal]

@nileshtiwaari Are you refering to Strucutred Streaming or DLT?
In case of Structured Streaming: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#streaming-deduplica...
About DLT, here's a thread from a couple of months ago https://community.databricks.com/t5/get-started-discussions/dlt-primary-key-deduplication-expectatio...