cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

"Detected a data update", what changed?

tom_shaffner
New Contributor III

In streaming flows I periodically get a "Detected a data update" error. This error generally seem to indicate that something has changed in the source table schema, but it's not immediately apparent what. In one case yesterday I pulled the source table and the destination table from the flow (which worked the day prior) and couldn't see any differences in the schemas. It's possible I missed something and need to write a programmatic comparison, but needing to do so seems to indicate an issue with the error message in my mind.

If I delete and recreate the tables and checkpoints the error will go away, but I can only do this so easily because I'm still in dev. Is there a way, particularly in a case where "wipe and restart" isn't such a great option, to see what is different about the data update that throws such an error? Or a way to add to the error message mention of what has changed?

1 REPLY 1

tom_shaffner
New Contributor III

@Kaniz Fatmaโ€‹ , Thanks, that helps. I was assuming this warning indicated a schema evolution, and based on what you say it likely wasn't and I just have to turn on IgnoreChanges any time I have a stream from a table that receives updates/upserts.

To be clear though, in such a case turning on IgnoreChanges would not prevent errors if I ever have a schema evolution, correct? I would still get a notification of such a change? Or will turning on IgnoreChanges mean I fail to get alerts on that also?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group