- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-10-2024 07:51 AM - last edited on 04-15-2024 08:26 AM by Retired_mod
Hello,
I'll complement @Retired_mod's answer, which is already very detailed.
First of all if you're considering Liquid Clustering, you need to know on which columns you'll cluster your table. These need to be the columns on which you're gonna merge.
Then, before running the MERGE you need to fetch only the rows that have changed or that are new : this is easily achieved with Change Data Feed.
Enable it for your table using :
ALTER TABLE my_table SET TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true');
Once you've got that, you can access the last modified rows with
SELECT * FROM table_changes('my_table', last_commit_version)
and use that dataset to run your merge, as explained in Kaniz's answer.
Let me know if that helped or if you need more details !