Raja_Databricks
New Contributor III

Hi @Retired_mod , @erigaud ,

Thank you for your assistance.

I've enabled Liquid Clustering on my Delta table, using the same columns as my merge keys. Daily, I load new data into a temporary table that's an exact structure of my main Delta table (also Liquid Clustered). Then, I merge the temporary data into the main table.

I have three questions:

  1. Before merging the temporary table into the main table, do I need to optimize the temporary table (Databricks suggests this in a hint)?
  2. Would Change Data Capture (CDC) be beneficial in this scenario?
  3. My Main Delta table will be highly growing one, will merge on primary key will be efficient over time?

Thanks in advance for your insights!