- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2022 01:31 AM
Hi All
I have a requirement to perform updates on a delta table that is the source for a streaming query.
I would like to be able to update the table and have the stream continue to work while also not ending up with duplicates.
From my research it seems that the ignoreDeletes option will not work as I am not going to be updating/deleting based on the partition column. The ignoreChanges option also looks unsuitable as it will generate duplicates of not only the rows I update, but also any other rows in the same files.
Does anyone have any suggestions/procedures they've used for similar in the past?
Thanks
- Labels:
-
Delta table
-
Source
-
Stream Processing
-
Table
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-13-2022 12:11 AM
Hi @Leszek
For your case ignoreChanges option will work, but you need to handle duplicates from your streaming app while writing to sink. If your sink is Delta then you can go with Delta Streaming Merge.
https://docs.databricks.com/_static/notebooks/merge-in-streaming.html
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-26-2022 03:21 AM
Maybe merging data from updated delta into next streaming delta will work?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-13-2022 12:11 AM
Hi @Leszek
For your case ignoreChanges option will work, but you need to handle duplicates from your streaming app while writing to sink. If your sink is Delta then you can go with Delta Streaming Merge.
https://docs.databricks.com/_static/notebooks/merge-in-streaming.html

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-19-2022 08:36 AM
Hey @Mathew Walters
Hope you are doing great.
Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution? Else please let us know if you need more help.
We'd love to hear from you.
Thanks!

