How to manage data reload in DLT

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

Hi, Community members

I had an situation to reload some data via DLT pipeline. All data are stored in landing storage account and they have been loaded in daily base. For example, from 1/Nov to 30/Nov.

For some reason, I need to reload the data of 25/Nov and I tried to use the following parameters to force the data relaod:

.option("cloudFiles.includeExistingFiles", includeExistingFiles)

.option("modifiedBefore",modifiedBefore)

.option("modifiedAfter",modifiedAfter)

However, no data are loaded or reloaded, even I deleted the data of the day in the bronze table. I guess it might because the checkpoint does not allow me to reload the data. and I end up with reload the data to a new schema which is not a desired outcome.

Could you please advise how I should manage the data reload scenario?

Thank you!