pavannaidu
Databricks Employee
Databricks Employee

VACUUM deletes untracked files under the UniForm metadata directory while preserving reachable Iceberg metadata. This is available starting DBR 17.2 and onwards. 

UniForm conversion performs snapshot expiration, but uses cleanExpiredFiles(false) by default. Therefore, OPTIMIZE and regular conversion only make old Iceberg metadata unreachable. They do not physically delete it. FULL VACUUM performs the physical cleanup after the delta.deletedFileRetentionDuration. All in all, if you have Preidctive Optimization enabled, this should automatically take care of Iceberg metadata cleanup. 

Few other suggestions to reduce metadata file size -- 

1/ Reduce Iceberg snapshot history 

'delta.universalformat.config.iceberg.history.expire.max-snapshot-age-ms' = '432000000' -- (default: milliseconds, or 5 days), it affects Iceberg snapshot history onlydelta.universalformat.config.iceberg.history.expire.min-snapshots-to-keep' = '1' -- minimum snapshots to keep

2/ Proactive metadata file deletion 

delta.universalformat.config.iceberg.write.metadata.delete-after-commit.enabled = true
delta.universalformat.config.iceberg.write.metadata.previous-versions-max = 100 -- will retain current metadata file along with 100 previous files

  3/ Gzip compression

delta.universalformat.config.iceberg.write.metadata.compression-codec = 'gzip' -- should reduce metadata file size, expected to be compatible with external Iceberg readers

 Suffice to say, test these suggestions on your workload to avoid unintended consequences. 

View solution in original post