Thank for the blog. Interesting read, I got some pointers that I need to check if we can implement that.

Question: why is a MERGE statement so 'stupid'? I am used to SQL Server behaviour, so when there is nothing to update, it will not update. However the Databricks MERGE will always update, regardless if rows were changed or not. This is annoying with a SCD2 approach and when the source is delivering overlapping delta windows.

The work-a-round for this is to compare first all records (and do a NULL check) and then decide if it can be forwarded to the MERGE, or implement a hash on the non-key fields and compare that. It is cumbersome and I'd rather have Databricks handle this plumbing.