cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Delta Live Table - Flow detected an update or delete to one or more rows in the source table

manish1987c
New Contributor III

I have create a pipeline where i am ingesting the data from bronze to silver and using SCD 1, however when i am trying to create gold table as dlt it is giving me error as "Flow 'user_silver' has FAILED fatally. An error occurred because we detected an update or delete to one or more rows in the source table."

#DLT  #DeltaLiveTables

manish1987c_0-1718341166099.png

 

 

Flow 'gold_table_test' has FAILED fatally. An error occurred because we detected an update or delete to one or more rows in the source table. Streaming tables may only use append-only streaming sources. If you expect to delete or update rows to the source table in the future, please convert table gold_table_test to a materialized view instead of a streaming table. If you are fine to skip changes, set the option 'skipChangeCommits' to 'true'. If you only have a one-time change in the source table and will only have append-only commits from now on, you can perform a full refresh to table gold_table_test to resolve the issue. A Full Refresh will attempt to clear all data from table gold_table_test and then load all data from the streaming source. The non-append change can be found at version 27. Operation: MERGE Username: mk1987c@outlook.com Source table name: silver_load_test 

 

manish1987c_1-1718341206991.png

 

 

 

 

 

 

 

 

2 REPLIES 2

shan_chandra
Databricks Employee
Databricks Employee

@manish1987c -The Streaming does not handle input that is not an append. you can set skipChangeCommits to true 

Can you please share some sample code... Like how to use it.. 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group