DLT Bronze: Incremental File Updates

[ChristianRRL]

Hi there, I would like to clarify if there's a way for bronze data to be ingested from "the same" CSV file if the file has been modified (i.e. new file with new records overwriting the old file)? Currently in my setup my bronze table is a `streaming table` and it successfully adds incremental data only as new files arrive, but if a file is overwritten with new data that file is ignored going forward.

Does DLT offer the capability to reprocess modified files?

[Lakshay · Accepted solution]

You can use the option "cloudFiles.allowOverwrites" in DLT. This option will allow you to read the same csv file again but you should use it cautiously, as it can lead to duplicate data being loaded.