Adding a New Column for Updated Date in Pipeline
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-26-2025 10:52 PM
I've successfully set up my pipeline and everything is working fine. I'd like to add a new column to our table that records the date whenver any records got updated. Could you advise on how to go about this?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-26-2025 11:20 PM
Hi @amarnathpal,
Do you want to add a new column after or while reading data from the source?
If you want to add after reading, you can just add a withcolumn statement after reading the data, but the data which has been processed already will have null values and the new incremental data will have the timestamp or date.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-26-2025 11:24 PM
Hi @nikhilj0421
yeah but that will give the current timestamp, but here i want based on the records got updated then i want to update the column with that day date.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-27-2025 12:25 AM
Hi @amarnathpal , could you pls elaborate your case?
But generally, if it's same as I think, you could use MERGE INTO or Delta Live Table to update the "updated" column.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-27-2025 12:25 AM
Do you want to add dates for the historical data as well?