urgent: apply changes into delta live tables
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-25-2023 03:08 AM - edited 11-25-2023 03:09 AM
Hi @Retired_mod , @Avnish_Jain
How can I implement cdc in sql dlt pipelines with a live table (not streaming) . I am trying to implement below where i am reading from external tables, loading data into bronze layer and then want to apply these changes into silver table but I am getting error with the apply changes into syntax.
create
or refresh live table bronze_dtl (
name string,
age string,
trk_nbr string,
perd_nbr string,
ld_timestamp timestamp
) comment "raw data" tblproperties (
'quality' = 'silver',
'delta.enableChangeDataFeed' = 'true'
) partitioned by (perd_nbr) as (
select
name,
age,
trk_nbr,
perd_nbr,
current_timestamp() as ld_timestamp
);
create
or refresh live table silver_dtl (
name string,
age string,
trk_nbr string,
perd_nbr string,
ld_timestamp timestamp
) comment "curated data" tblproperties (
'quality' = 'silver',
'delta.enableChangeDataFeed' = 'true'
) partitioned by (perd_nbr);
apply changes into live.silver_dtl
from
live.bronze_dtl keys (trk_nbr, perd_nbr) sequence by ld_timestamp stored as scd type 1;
Please let me know wheres the issue, and any shortcomings in the above example specifically in cdc part