I have a scenario to implement using the delta live tables.
I get the id and timestamp column from source and I have to load that into my delta live streaming output table only if the source timestamp for less that the existing value in the output table.
I am getting an error mentioning related to the output table -> "Dataset is defined in the pipeline but could not be resolved"
I am performing the following operations in my notebook.
CREATE LIVE STREAMING VIEW dlt_src_table AS SELECT id,created_dt FROM STREAM(delta table)
CREATE STREAMING LIVE TABLE output
( id INT,
src_min_time TIMESTAMP
)
CREATE TEMPORARY STREAMING LIVE TABLE tmp
AS
SELECT id, created_dt AS src_min_time
FROM STREAM(LIVE.dlt_src_table) src
LEFT JOIN LIVE.Output op
ON src.id = op.id
WHERE op.id IS NULL OR src.created_dt < op.src_min_time
APPLY CHANGES INTO LIVE.output
FROM STREAM(LIVE.tmp)
KEYS(id)
SEQUENCE BY src_min_time
STORED AS SCD TYPE 1
Thanks for your help in advance