Delta Live tables - Refer the output table to load by checking condition for scd type 1

[karthikus]

I have a scenario to implement using the delta live tables.

I get the id and timestamp column from source and I have to load that into my delta live streaming output table only if the source timestamp for less that the existing value in the output table.

I am getting an error mentioning related to the output table  -> "Dataset is defined in the pipeline but could not be resolved"

I am performing the following operations in my notebook.

CREATE LIVE STREAMING VIEW dlt_src_table AS SELECT id,created_dt FROM STREAM(delta table)

CREATE STREAMING LIVE TABLE output

(   id INT, 

src_min_time TIMESTAMP

)

CREATE TEMPORARY STREAMING LIVE TABLE  tmp

AS

SELECT id, created_dt AS src_min_time

FROM STREAM(LIVE.dlt_src_table) src

LEFT JOIN LIVE.Output op

ON src.id = op.id

WHERE  op.id IS NULL OR src.created_dt < op.src_min_time

APPLY CHANGES INTO LIVE.output

FROM STREAM(LIVE.tmp)

KEYS(id)

SEQUENCE BY src_min_time

STORED AS SCD TYPE 1

Thanks for your help in advance

[Kaniz]

Hi @karthikus, 

• Double-check the table and column names, ensuring they match your actual Delta tables.
• Verify that the schema of the output table matches the expected schema.
• Confirm that the Delta Live Tables environment is set up correctly.