urgent: apply changes into delta live tables

Faisal
Contributor

Hi @Retired_mod , @Avnish_Jain 

How can I implement cdc in sql dlt pipelines with a live table (not streaming) . I am trying to implement below where i am reading from external tables, loading data into bronze layer and then want to apply these changes into silver table but I am getting error with the apply changes into syntax.

 

create
or refresh live table bronze_dtl (
  name string,
  age string,
  trk_nbr string,
  perd_nbr string,
  ld_timestamp timestamp
) comment "raw data" tblproperties (
  'quality' = 'silver',
  'delta.enableChangeDataFeed' = 'true'
) partitioned by (perd_nbr) as (
  select 
  name,
  age,
  trk_nbr,
  perd_nbr,
  current_timestamp() as ld_timestamp
);

create
or refresh live table silver_dtl (
  name string,
  age string,
  trk_nbr string,
  perd_nbr string,
  ld_timestamp timestamp
) comment "curated data" tblproperties (
  'quality' = 'silver',
  'delta.enableChangeDataFeed' = 'true'
) partitioned by (perd_nbr);

apply changes into live.silver_dtl
from
  live.bronze_dtl keys (trk_nbr, perd_nbr) sequence by ld_timestamp stored as scd type 1;

 

 Please let me know wheres the issue, and any shortcomings in the above example specifically in cdc part