cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Delta table - version number change on merge

SKC01
New Contributor II

I am running a merge with pyspark on a delta table in which nothing is getting updated in the target table. Still target table version is incremented when I check the table history. Is that expected behavior?

1 REPLY 1

Sidhant07
Databricks Employee
Databricks Employee

Yes, this is the expected behavior. In Delta Lake, every operation, including MERGE, is atomic. This means that each operation is a transaction that can either succeed completely or fail; it cannot have partial success. Even if the MERGE operation doesn't result in any changes to the target table, it is still considered a transaction and thus increments the table version.When a MERGE operation is performed, Delta Lake performs several steps:1. It identifies the rows in the source data that match the condition specified in the MERGE statement.
2. It applies the update, delete, or insert actions to the matched and not-matched source and target rows.
3. It writes out the result as a new version of the target Delta table.Even if no rows are updated, deleted, or inserted, these steps are still performed, and the result is written out as a new version of the target Delta table. This is why the table version is incremented even if nothing is updated in the target table.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group