How to manage data reload in DLT

harvey-c — Thu, 30 Nov 2023 00:58:22 GMT

Hi, Community members

I had an situation to reload some data via DLT pipeline. All data are stored in landing storage account and they have been loaded in daily base. For example, from 1/Nov to 30/Nov.

For some reason, I need to reload the data of 25/Nov and I tried to use the following parameters to force the data relaod:

.option("cloudFiles.includeExistingFiles", includeExistingFiles)

.option("modifiedBefore",modifiedBefore)

.option("modifiedAfter",modifiedAfter)

However, no data are loaded or reloaded, even I deleted the data of the day in the bronze table. I guess it might because the checkpoint does not allow me to reload the data. and I end up with reload the data to a new schema which is not a desired outcome.

Could you please advise how I should manage the data reload scenario?

Thank you!

topic How to manage data reload in DLT in Data Engineering

How to manage data reload in DLT