Delta table - version number change on merge

SKC01 — Sat, 07 Oct 2023 21:31:11 GMT

I am running a merge with pyspark on a delta table in which nothing is getting updated in the target table. Still target table version is incremented when I check the table history. Is that expected behavior?

Re: Delta table - version number change on merge

Sidhant07 — Thu, 09 Nov 2023 10:59:00 GMT

Yes, this is the expected behavior. In Delta Lake, every operation, including MERGE, is atomic. This means that each operation is a transaction that can either succeed completely or fail; it cannot have partial success. Even if the MERGE operation doesn't result in any changes to the target table, it is still considered a transaction and thus increments the table version.When a MERGE operation is performed, Delta Lake performs several steps:1. It identifies the rows in the source data that match the condition specified in the MERGE statement.
2. It applies the update, delete, or insert actions to the matched and not-matched source and target rows.
3. It writes out the result as a new version of the target Delta table.Even if no rows are updated, deleted, or inserted, these steps are still performed, and the result is written out as a new version of the target Delta table. This is why the table version is incremented even if nothing is updated in the target table.

topic Re: Delta table - version number change on merge in Data Engineering

Delta table - version number change on merge

Re: Delta table - version number change on merge