topic Re: Adding a New Column for Updated Date in Pipeline in Data Engineering

Adding a New Column for Updated Date in Pipeline

amarnathpal — Tue, 27 May 2025 05:52:53 GMT

I've successfully set up my pipeline and everything is working fine. I'd like to add a new column to our table that records the date whenver any records got updated. Could you advise on how to go about this?

Re: Adding a New Column for Updated Date in Pipeline

nikhilj0421 — Tue, 27 May 2025 06:20:53 GMT

Hi @amarnathpal,

Do you want to add a new column after or while reading data from the source?

If you want to add after reading, you can just add a withcolumn statement after reading the data, but the data which has been processed already will have null values and the new incremental data will have the timestamp or date.

Re: Adding a New Column for Updated Date in Pipeline

amarnathpal — Tue, 27 May 2025 06:24:43 GMT

Hi @nikhilj0421

yeah but that will give the current timestamp, but here i want based on the records got updated then i want to update the column with that day date.

Re: Adding a New Column for Updated Date in Pipeline

canada2k1 — Tue, 27 May 2025 07:25:01 GMT

Hi @amarnathpal , could you pls elaborate your case?

But generally, if it's same as I think, you could use MERGE INTO or Delta Live Table to update the "updated" column.

Re: Adding a New Column for Updated Date in Pipeline

nikhilj0421 — Tue, 27 May 2025 07:25:55 GMT

Do you want to add dates for the historical data as well?