3 weeks ago
How to remove duplicates in streaming query on the basis of some id?
@nileshtiwaari Are you refering to Strucutred Streaming or DLT?In case of Structured Streaming: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#streaming-deduplica...About DLT, here's a thread from a couple of months ago https://community.databricks.com/t5/get-started-discussions/dlt-primary-key-deduplication-expectatio...
never-displayed
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!