Streaming data apply change error not function with incoming files

[Data_Sam]

Hi all,

When I design a streaming data pipeline with incoming moving files and used apply chnge function on silver table comparing change between bronze and silver for removing duplicates based on key columns, do you know why I got ignore change to true error?

The flow is like this:

Files - Stream table bronze raw - temp bronze - apply change function to create bronze table - temp table - silver table.

The error happend on the apply change function when more files are moving into the folder.

The only solution is to refresh all. If you do refresh but not all, It will pop up ignore changes... change a different directory.

Thank

[Anonymous]

@Raymond Huang​ :

The error message "ignore changes to true" typically occurs when you are trying to apply changes to a table using Delta Lake's change data capture (CDC) feature, but you have set the option ignoreChanges to true. This option tells Delta Lake to skip any changes that it cannot understand or apply to the table. It's possible that the error is occurring because the change data being applied to the table is not in a format that Delta Lake can understand, or there may be some other issue with the data. Refreshing the table will clear the metadata and cause Delta Lake to reprocess all the data in the table from scratch, so this may resolve the issue.

However, it's important to note that refreshing the table can be an expensive operation, especially if the table contains a large amount of data. It's also possible that refreshing the table may not always resolve the issue, and you may need to investigate further to determine the root cause of the problem.

One way to investigate the issue further would be to look at the logs and error messages generated by the apply change function to see if there are any specific errors or warnings that are being raised. You could also try disabling the ignoreChanges option to see if this allows the changes to be applied successfully.