Restore Delta table after adding columns.

[B_Seibert]

At version 3 of our Delta Lake table we added a column. We later restored from version 11 back to version 10, which is now the most current version. But now when we run the table build from Azure Data Factory (ADF) on the full history of the data, we get an error at version 4 saying: "Detected deleted data from streaming source"

I think that if we want to do a RESTORE, we can not then go back and run our ingestion on the same ADLS directory which includes both the 0 to 3 versions of .csv files with one less column, as well as, the 4 to 10 versions that have one more column.

RESTORE TABLE table_name TO TIMESTAMP AS OF '2022-10-31 00:00:00';

ADD COLUMNS
{"columns": "[{\"column\":{\"name\":\"my_column_name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}}]"}

[Debayan]

Hi, Could you please ignoreDeletes. Please refer: https://docs.databricks.com/structured-streaming/delta-lake.html#ignore-updates-and-deletes

Please let us know if this helps.

[B_Seibert]

ignoreDeletes works. But I recommend to other developers that you have a think aboout all of the schema change scenarios and solve this problem above as part of a complete solution to every schema change scenario, instead of dealing with it as a one-off.