Hi @Walter_C ,
Thanks for your suggestion.

Can you help me with all the possible failure scenario need to be handled while doing incremental load ? Gold table is delta only. The data coming into Gold table is from DLT silver table. We are not suppose to use the DLT in the gold layer because of complex transformations. We have create a single pyspark notebook to read the silver table and then extracted the incremental data from it based on last run timestamp. Now performing the transformation on the dataframe . Once this is done then loading the all three final objects. We have a single task so repair run is not possible here.