NandiniN
Databricks Employee
Databricks Employee

Hi @tenzinpro , 

This is an expected error. "DELTA_SOURCE_TABLE_IGNORE_CHANGES] Detected a data update"

As explained in the error: This is currently not supported. If this is going to happen regularly and you are okay to skip changes, set the option 'skipChangeCommits' to 'true'. If you would like the data update to be reflected, please restart this query with a fresh checkpoint directory or do a full refresh if you are using DLT. If you need to handle these changes, please switch to MV.

I recommended below options to the customer to achieve their use case:1. Define a Materialized View (MV) instead of Streaming table (ST) or2. Use skipChangeCommits to skip the changes happened on the source like updates/deletes or3. Use the source with CDF and follow the steps here:https://community.databricks.com/t5/technical-blog/propagating-deletes-managing-data-removal-using-d...

And Incremental refresh for materialized views as suggested by @BS_THE_ANALYST is the best way.

View solution in original post