Hi All
I have a requirement to perform updates on a delta table that is the source for a streaming query.
I would like to be able to update the table and have the stream continue to work while also not ending up with duplicates.
From my research it seems that the ignoreDeletes option will not work as I am not going to be updating/deleting based on the partition column. The ignoreChanges option also looks unsuitable as it will generate duplicates of not only the rows I update, but also any other rows in the same files.
Does anyone have any suggestions/procedures they've used for similar in the past?
Thanks