https://community.databricks.com/t5/get-started-discussions/streaming-query/m-p/72740#M7436 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/105222">@nileshtiwaari</a>&nbsp;Are you refering to Strucutred Streaming or DLT?<BR />In case of Structured Streaming:&nbsp;<A href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#streaming-deduplication" target="_blank">https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#streaming-deduplication</A><BR />About DLT, here's a thread from a couple of months ago&nbsp;<A href="https://community.databricks.com/t5/get-started-discussions/dlt-primary-key-deduplication-expectations-vs-constraints-vs/td-p/58290" target="_blank">https://community.databricks.com/t5/get-started-discussions/dlt-primary-key-deduplication-expectations-vs-constraints-vs/td-p/58290</A></P> Wed, 12 Jun 2024 06:56:25 GMT daniel_sahal 2024-06-12T06:56:25Z https://community.databricks.com/t5/get-started-discussions/streaming-query/m-p/72711#M7435 <P>How to remove duplicates in streaming query on the basis of some id?</P> Wed, 12 Jun 2024 04:51:41 GMT https://community.databricks.com/t5/get-started-discussions/streaming-query/m-p/72711#M7435 nileshtiwaari 2024-06-12T04:51:41Z https://community.databricks.com/t5/get-started-discussions/streaming-query/m-p/72740#M7436 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/105222">@nileshtiwaari</a>&nbsp;Are you refering to Strucutred Streaming or DLT?<BR />In case of Structured Streaming:&nbsp;<A href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#streaming-deduplication" target="_blank">https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#streaming-deduplication</A><BR />About DLT, here's a thread from a couple of months ago&nbsp;<A href="https://community.databricks.com/t5/get-started-discussions/dlt-primary-key-deduplication-expectations-vs-constraints-vs/td-p/58290" target="_blank">https://community.databricks.com/t5/get-started-discussions/dlt-primary-key-deduplication-expectations-vs-constraints-vs/td-p/58290</A></P> Wed, 12 Jun 2024 06:56:25 GMT https://community.databricks.com/t5/get-started-discussions/streaming-query/m-p/72740#M7436 daniel_sahal 2024-06-12T06:56:25Z