This is what we are using but I was wondering if this is the most efficient way.

In our example, we get around 30-40 million records with every new file (source is giving the complete snapshot of data). Thus, even if there is change in a few records, it need to go through merging of all the 30-40 million records, which takes some time.

Not sure if there is any efficient way to handle this complete override of data use case without the expensive merge.