What gets overridden when writing overriding a delta lake table?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-13-2022 07:02 AM
I just tried to write to a delta lake table using override mode, and I found that history is reserved. It's unclear to me how the data is overridden, and how long the history could be preserved.
As they say, a code is better than a thousand words:
mytable.write.mode("overwrite").saveAsTable("apps")
%sql
SELECT id,display_name FROM apps WHERE id = 3;
id | name
3 | old
# mytabl gets updated
mytable.write.mode("overwrite").saveAsTable("apps")
%sql
SELECT id,display_name FROM apps WHERE id = 3;
id | name
3 | new
%sql
SELECT id,display_name FROM apps VERSION as OF 0 WHERE id = 3;
id | name
3 | old
Also, what is the performance and cost implication of having many versions of a single table?
Labels:
- Labels:
-
Delta lake table