Hi everyone! ๐
In my previous post, I discussed how enabling Deletion Vectors helped reduce our Delta MERGE runtime from 22 minutes down to 6 minutes by eliminating write amplification.
However, deferring file rewrites introduces an important architectural consideration: Logical deletion is not physical deletion.
While Deletion Vectors mark rows as deleted logically, the underlying Parquet files still contain those rows physically. To maintain an efficient Lakehouse, it is critical to understand how different maintenance operations interact:
Deletion Vectors: Soft-delete rows logically to avoid immediate file rewrites during write operations.
REORG TABLE ... APPLY (PURGE): Physically rewrites files to apply deletion-vector changes and remove deleted rows.
VACUUM: Removes old, unreferenced files that fall outside the table retention window.
I wrote a complete breakdown covering why running PURGE immediately after every MERGE defeats the purpose of Deletion Vectors and how to structure your maintenance routines effectively.
You can read the full deep dive here:
๐ Deletion Vectors Donโt Delete The Data The Way You Think They Do
How are you currently managing table maintenance and REORG PURGE schedules on your high-frequency MERGE workloads? Would love to hear how other teams handle this balance!
