When running a Merge, if records from the table are deleted are the underlying files that contain the records deleted as well?

User16826992666 — Wed, 16 Jun 2021 16:38:39 GMT

I know I have the option to delete rows from a Delta table when running a merge. But I'm confused about how that would actually affect the files that contain the deleted records. Are those files deleted, or are they rewritten, or what?

Re: When running a Merge, if records from the table are deleted are the underlying files that contain the records deleted as well?

sajith_appukutt — Fri, 18 Jun 2021 20:36:06 GMT

Delta implements MERGE by physically rewriting existing files.

It is implemented in two steps.

Perform an inner join between the target table and source table to select all files that have matches.
Perform an outer join between the selected files in the target and source tables and write out the updated/deleted/inserted data.

Here is an article that explain the DML internals of delta - https://databricks.com/blog/2020/09/29/diving-into-delta-lake-dml-internals-update-delete-merge.html

The old files that are not referenced anymore would get garbage collected eventually when you run VACUUM

topic When running a Merge, if records from the table are deleted are the underlying files that contain the records deleted as well? in Data Engineering

When running a Merge, if records from the table are deleted are the underlying files that contain the records deleted as well?

Re: When running a Merge, if records from the table are deleted are the underlying files that contain the records deleted as well?