topic Re: Dealing with updates to a delta table being used as a streaming source in Data Engineering

Dealing with updates to a delta table being used as a streaming source

Confused — Tue, 11 Jan 2022 09:31:42 GMT

Hi All

I have a requirement to perform updates on a delta table that is the source for a streaming query.

I would like to be able to update the table and have the stream continue to work while also not ending up with duplicates.

From my research it seems that the ignoreDeletes option will not work as I am not going to be updating/deleting based on the partition column. The ignoreChanges option also looks unsuitable as it will generate duplicates of not only the rows I update, but also any other rows in the same files.

Does anyone have any suggestions/procedures they've used for similar in the past?

Thanks

Re: Dealing with updates to a delta table being used as a streaming source

Leszek — Wed, 26 Jan 2022 11:21:16 GMT

Maybe merging data from updated delta into next streaming delta will work?

https://www.youtube.com/watch?v=2Iy5S0Hf4XM

Re: Dealing with updates to a delta table being used as a streaming source

Manjunath — Sun, 13 Mar 2022 08:11:38 GMT

Hi @Leszek

For your case ignoreChanges option will work, but you need to handle duplicates from your streaming app while writing to sink. If your sink is Delta then you can go with Delta Streaming Merge.

https://docs.databricks.com/_static/notebooks/merge-in-streaming.html

Re: Dealing with updates to a delta table being used as a streaming source

Anonymous — Thu, 19 May 2022 15:36:37 GMT

Hey @Mathew Walters

Hope you are doing great.

Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!