cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Restore Delta table after adding columns.

B_Seibert
New Contributor III

At version 3 of our Delta Lake table we added a column. We later restored from version 11 back to version 10, which is now the most current version. But now when we run the table build from Azure Data Factory (ADF) on the full history of the data, we get an error at version 4 saying: "Detected deleted data from streaming source"

I think that if we want to do a RESTORE, we can not then go back and run our ingestion on the same ADLS directory which includes both the 0 to 3 versions of .csv files with one less column, as well as, the 4 to 10 versions that have one more column.

RESTORE TABLE table_name TO TIMESTAMP AS OF '2022-10-31 00:00:00';

ADD COLUMNS
{"columns": "[{\"column\":{\"name\":\"my_column_name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}}]"}

3 REPLIES 3

Debayan
Esteemed Contributor III
Esteemed Contributor III

Hi, Could you please ignoreDeletes. Please refer: https://docs.databricks.com/structured-streaming/delta-lake.html#ignore-updates-and-deletes

Please let us know if this helps.

Kaniz
Community Manager
Community Manager

Hi @Brant Seibert​, We haven’t heard from you on the last response from @Debayan Mukherjee​​, and I was checking back to see if his suggestions helped you.

Or else, If you have any solution, please do share that with the community as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

B_Seibert
New Contributor III

ignoreDeletes works. But I recommend to other developers that you have a think aboout all of the schema change scenarios and solve this problem above as part of a complete solution to every schema change scenario, instead of dealing with it as a one-off.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.