"Detected a data update", what changed?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ05-18-2022 09:14 AM
In streaming flows I periodically get a "Detected a data update" error. This error generally seem to indicate that something has changed in the source table schema, but it's not immediately apparent what. In one case yesterday I pulled the source table and the destination table from the flow (which worked the day prior) and couldn't see any differences in the schemas. It's possible I missed something and need to write a programmatic comparison, but needing to do so seems to indicate an issue with the error message in my mind.
If I delete and recreate the tables and checkpoints the error will go away, but I can only do this so easily because I'm still in dev. Is there a way, particularly in a case where "wipe and restart" isn't such a great option, to see what is different about the data update that throws such an error? Or a way to add to the error message mention of what has changed?
- Labels:
-
Delta Tables
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ05-19-2022 10:33 AM
@Kaniz Fatmaโ , Thanks, that helps. I was assuming this warning indicated a schema evolution, and based on what you say it likely wasn't and I just have to turn on IgnoreChanges any time I have a stream from a table that receives updates/upserts.
To be clear though, in such a case turning on IgnoreChanges would not prevent errors if I ever have a schema evolution, correct? I would still get a notification of such a change? Or will turning on IgnoreChanges mean I fail to get alerts on that also?

