Hello !
We're currently building a pipeline of file ingestion using a Delta Live Tables pipeline and autoloader.
The bronze tables are pretty much the following schema :
file_name | file_upload_date | colA | colB
(Well, there are actually 250+ columns but you get the idea)
The bronze table is append only, with possibly some duplicates because some files can be uploaded several times with corrections, but they will have the same name. The logic I'm trying to implement table is the following:
- A file is loaded in bronze, lets say 500 rows with file_name = file_name_A.csv and the corresponding upload_date (that part is fine, just standard auto-loader)
- In silver we already had some rows (lets say 1000) for that file_name, but an older upload_date. In that case we want to replace all the 1000 rows by the newer 500 rows.
How would someone go about doing something like this using Delta Live Table ?
Thank you !