Hi, Community members
I had an situation to reload some data via DLT pipeline. All data are stored in landing storage account and they have been loaded in daily base. For example, from 1/Nov to 30/Nov.
For some reason, I need to reload the data of 25/Nov and I tried to use the following parameters to force the data relaod:
.option("cloudFiles.includeExistingFiles", includeExistingFiles)
.option("modifiedBefore",modifiedBefore)
.option("modifiedAfter",modifiedAfter)
However, no data are loaded or reloaded, even I deleted the data of the day in the bronze table. I guess it might because the checkpoint does not allow me to reload the data. and I end up with reload the data to a new schema which is not a desired outcome.
Could you please advise how I should manage the data reload scenario?
Thank you!
