Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-10-2024 11:37 PM
Hi @Retired_mod , @erigaud ,
Thank you for your assistance.
I've enabled Liquid Clustering on my Delta table, using the same columns as my merge keys. Daily, I load new data into a temporary table that's an exact structure of my main Delta table (also Liquid Clustered). Then, I merge the temporary data into the main table.
I have three questions:
- Before merging the temporary table into the main table, do I need to optimize the temporary table (Databricks suggests this in a hint)?
- Would Change Data Capture (CDC) be beneficial in this scenario?
- My Main Delta table will be highly growing one, will merge on primary key will be efficient over time?
Thanks in advance for your insights!