Streaming Query
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-11-2024 09:51 PM
How to remove duplicates in streaming query on the basis of some id?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-11-2024 11:56 PM
@nileshtiwaari Are you refering to Strucutred Streaming or DLT?
In case of Structured Streaming: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#streaming-deduplica...
About DLT, here's a thread from a couple of months ago https://community.databricks.com/t5/get-started-discussions/dlt-primary-key-deduplication-expectatio...
