Delta Live tables - Refer the output table to load by checking condition for scd type 1

kmaley — Mon, 11 Mar 2024 15:42:40 GMT

I have a scenario to implement using the delta live tables.

I get the id and timestamp column from source and I have to load that into my delta live streaming output table only if the source timestamp for less that the existing value in the output table.

I am getting an error mentioning related to the output table -> "Dataset is defined in the pipeline but could not be resolved"

I am performing the following operations in my notebook.

CREATE LIVE STREAMING VIEW dlt_src_table AS SELECT id,created_dt FROM STREAM(delta table)

CREATE STREAMING LIVE TABLE output

( id INT,

src_min_time TIMESTAMP

)

CREATE TEMPORARY STREAMING LIVE TABLE tmp

SELECT id, created_dt AS src_min_time

FROM STREAM(LIVE.dlt_src_table) src

LEFT JOIN LIVE.Output op

ON src.id = op.id

WHERE op.id IS NULL OR src.created_dt < op.src_min_time

APPLY CHANGES INTO LIVE.output

FROM STREAM(LIVE.tmp)

KEYS(id)

SEQUENCE BY src_min_time

STORED AS SCD TYPE 1

Thanks for your help in advance

topic Delta Live tables - Refer the output table to load by checking condition for scd type 1 in Data Engineering

Delta Live tables - Refer the output table to load by checking condition for scd type 1