@szymon_dybczak i could resolve it now! basically, i broke the process down into further subprocesses, for each sub process, i cached and wrote them all into delta table (without overwritting), the next subprocess needs to read data in the delta table written by the prev subprocess. that way i could write all the data into the delta table without losing any records. so glad the pain has ended now!