Adding a New Column for Updated Date in Pipeline

[amarnathpal]

I've successfully set up my pipeline and everything is working fine. I'd like to add a new column to our table that records the date whenver any records got updated. Could you advise on how to go about this?

[nikhilj0421]

Hi @amarnathpal, 

Do you want to add a new column after or while reading data from the source?

If you want to add after reading, you can just add a withcolumn statement after reading the data, but the data which has been processed already will have null values and the new incremental data will have the timestamp or date. 

[amarnathpal]

Hi @nikhilj0421 

yeah but that will give the current timestamp, but here i want based on the records got updated then i want to update the column with that day date.

[canada2k1]

Hi @amarnathpal , could you pls elaborate your case?

But generally, if it's same as I think, you could use MERGE INTO or Delta Live Table to update the "updated" column.

[nikhilj0421]

Do you want to add dates for the historical data as well?