Databricks Community

User16826992666 · ‎06-16-2021

I know I have the option to delete rows from a Delta table when running a merge. But I'm confused about how that would actually affect the files that contain the deleted records. Are those files deleted, or are they rewritten, or what?

sajith_appukutt · ‎06-18-2021

Delta implements MERGE by physically rewriting existing files.

It is implemented in two steps.

Perform an inner join between the target table and source table to select all files that have matches.
Perform an outer join between the selected files in the target and source tables and write out the updated/deleted/inserted data.

Here is an article that explain the DML internals of delta - https://databricks.com/blog/2020/09/29/diving-into-delta-lake-dml-internals-update-delete-merge.html

The old files that are not referenced anymore would get garbage collected eventually when you run VACUUM

View solution in original post

sajith_appukutt · ‎06-18-2021

Delta implements MERGE by physically rewriting existing files.

It is implemented in two steps.

Perform an inner join between the target table and source table to select all files that have matches.
Perform an outer join between the selected files in the target and source tables and write out the updated/deleted/inserted data.

Here is an article that explain the DML internals of delta - https://databricks.com/blog/2020/09/29/diving-into-delta-lake-dml-internals-update-delete-merge.html

The old files that are not referenced anymore would get garbage collected eventually when you run VACUUM