"Detected a data update", what changed?

tom_shaffner — Wed, 18 May 2022 16:14:18 GMT

In streaming flows I periodically get a "Detected a data update" error. This error generally seem to indicate that something has changed in the source table schema, but it's not immediately apparent what. In one case yesterday I pulled the source table and the destination table from the flow (which worked the day prior) and couldn't see any differences in the schemas. It's possible I missed something and need to write a programmatic comparison, but needing to do so seems to indicate an issue with the error message in my mind.

If I delete and recreate the tables and checkpoints the error will go away, but I can only do this so easily because I'm still in dev. Is there a way, particularly in a case where "wipe and restart" isn't such a great option, to see what is different about the data update that throws such an error? Or a way to add to the error message mention of what has changed?

Re: "Detected a data update", what changed?

tom_shaffner — Thu, 19 May 2022 17:33:07 GMT

@Kaniz Fatma , Thanks, that helps. I was assuming this warning indicated a schema evolution, and based on what you say it likely wasn't and I just have to turn on IgnoreChanges any time I have a stream from a table that receives updates/upserts.

To be clear though, in such a case turning on IgnoreChanges would not prevent errors if I ever have a schema evolution, correct? I would still get a notification of such a change? Or will turning on IgnoreChanges mean I fail to get alerts on that also?

topic Re: "Detected a data update", what changed? in Data Engineering

"Detected a data update", what changed?

Re: "Detected a data update", what changed?