Restoring a table from a Delta live pipeline
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-01-2025 07:28 PM
I have a DLT pipeline running to ingest files from storage using autoloader. We have a bronze table and a Silver table.
A question came up from the team on how to restore DLT tables to a previous version in case of some incorrect transformation.
When we try to execute the RESTORE to version command we get the below error.
[STREAMING_TABLE_OPERATION_NOT_ALLOWED.UNSUPPORTED_OPERATION] The operation RESTORE is not allowed: The operation is not supported on Streaming Tables. SQLSTATE: 42601
Then I tried deleting some records from the Silver table and restart the pipeline. Since the tables are streaming, the deleted records are not repopulated from silver.
So my questions are :
1) What is the best way to restore a DLT table to its previous version and do a backfill or rerun of the last DLT run?
2) Do we have to do a full refresh to accomplish this?