- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a month ago
VACUUM deletes untracked files under the UniForm metadata directory while preserving reachable Iceberg metadata. This is available starting DBR 17.2 and onwards.
UniForm conversion performs snapshot expiration, but uses cleanExpiredFiles(false) by default. Therefore, OPTIMIZE and regular conversion only make old Iceberg metadata unreachable. They do not physically delete it. FULL VACUUM performs the physical cleanup after the delta.deletedFileRetentionDuration. All in all, if you have Preidctive Optimization enabled, this should automatically take care of Iceberg metadata cleanup.
Few other suggestions to reduce metadata file size --
1/ Reduce Iceberg snapshot history
'delta.universalformat.config.iceberg.history.expire.max-snapshot-age-ms' = '432000000' -- (default: milliseconds, or 5 days), it affects Iceberg snapshot history onlydelta.universalformat.config.iceberg.history.expire.min-snapshots-to-keep' = '1' -- minimum snapshots to keep
2/ Proactive metadata file deletion
delta.universalformat.config.iceberg.write.metadata.delete-after-commit.enabled = true delta.universalformat.config.iceberg.write.metadata.previous-versions-max = 100 -- will retain current metadata file along with 100 previous files
3/ Gzip compression
delta.universalformat.config.iceberg.write.metadata.compression-codec = 'gzip' -- should reduce metadata file size, expected to be compatible with external Iceberg readers
Suffice to say, test these suggestions on your workload to avoid unintended consequences.