Hello Databricks community,

I'm working on a pipeline and would like to implement a common use case using Delta Live Tables. The pipeline should include the following steps:

1. Incrementally load data from Table A as a batch.
2. If the pipeline has previously failed, include older batches that were not processed due to the failure.
3. Perform some transformations or processing on the data.
4. Write the output to a destination table.

The motivation behind this implementation is to handle new data as a Spark batch because Spark Streaming does not support many commonly required aggregations. Additionally, this approach is intended to handle pipeline failures that may arise due to new deployments or unexpected changes in the data. These changes can potentially break the transformations or processing on the data, resulting in downtime. After deploying fixes, the pipeline should recover by loading and processing failed batches without recomputing everything historically. This recovery mechanism helps avoid incurring huge costs when dealing with large volumes of data.

I am seeking guidance on the best practices for implementing this scenario using Delta Live Tables. In particular, how can I ensure that the pipeline correctly handles previously failed batches and processes them along with new data, while also providing a robust recovery mechanism?

Any help or insights would be greatly appreciated!

Thank you in advance!