topic Restore Delta table after adding columns. in Data Engineering

Restore Delta table after adding columns.

B_Seibert — Mon, 07 Nov 2022 22:52:42 GMT

At version 3 of our Delta Lake table we added a column. We later restored from version 11 back to version 10, which is now the most current version. But now when we run the table build from Azure Data Factory (ADF) on the full history of the data, we get an error at version 4 saying: "Detected deleted data from streaming source"

I think that if we want to do a RESTORE, we can not then go back and run our ingestion on the same ADLS directory which includes both the 0 to 3 versions of .csv files with one less column, as well as, the 4 to 10 versions that have one more column.

RESTORE TABLE table_name TO TIMESTAMP AS OF '2022-10-31 00:00:00';

ADD COLUMNS
{"columns": "[{\"column\":{\"name\":\"my_column_name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}}]"}

Re: Restore Delta table after adding columns.

B_Seibert — Mon, 21 Nov 2022 12:26:27 GMT

ignoreDeletes works. But I recommend to other developers that you have a think aboout all of the schema change scenarios and solve this problem above as part of a complete solution to every schema change scenario, instead of dealing with it as a one-off.

Re: Restore Delta table after adding columns.

Debayan — Wed, 09 Nov 2022 07:27:15 GMT

Hi, Could you please ignoreDeletes. Please refer: https://docs.databricks.com/structured-streaming/delta-lake.html#ignore-updates-and-deletes

Please let us know if this helps.