- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-13-2025 06:35 AM
Here’s how to approach cleaning and maintaining Apache Iceberg metadata on Databricks, and how it differs from Delta workflows.
First, know your table type
-
For Unity Catalog–managed Iceberg tables, Databricks runs table maintenance for you (predictive optimization) — including snapshot expiration and orphan-file cleanup — so you rarely need to run these actions manually.
-
For foreign/external Iceberg tables (or if you intentionally disable automation), you may choose to run specific Iceberg maintenance procedures yourself.
Action-by-action guidance
expireSnapshots
-
Yes — expireSnapshots is recommended to bound your time-travel/rollback window and keep metadata compact. On managed Iceberg, UC automates snapshot expiration; choose manual retention only when you need tighter control.
-
Don’t assume the same retention as your Delta VACUUM. Set Iceberg’s retention to match your operational needs (time travel, audit requirements, longest-running jobs), independent of Delta’s retention checks. If you do run it manually, you can use Iceberg procedures, for example:
SQL (Iceberg proc)
CALL <catalog>.system.expire_snapshots(table => 'db.tbl', older_than => CURRENT_TIMESTAMP - INTERVAL 7 DAYS);or (client-dependent syntax)
ALTER TABLE db.tbl EXECUTE expire_snapshots(retention_threshold => '7d');
deleteOrphanFiles
-
Only run deleteOrphanFiles when the table’s storage location is used exclusively by Iceberg and you’re certain those files aren’t referenced elsewhere. If the same Parquet files serve multiple formats (e.g., Delta with Iceberg reads/UniForm), deleting “orphans” from Iceberg’s perspective can break Delta readers that still reference them. In short: not safe if Delta still references those files.
Why: Databricks supports workflows where a single copy of Parquet data is served to multiple formats; removing files because they’re “unreferenced” in Iceberg can invalidate concurrent readers in Delta or path-based Iceberg clients until metadata is refreshed.
rewriteManifests
-
rewriteManifests is safe and often beneficial — it rewrites manifest files for planning efficiency and creates a new snapshot (data remains unchanged). On managed Iceberg, UC periodically optimizes metadata for you; consider manual rewrites for external tables or after heavy streaming/append workloads that produce many small manifests.
-
Practical tips (when you run it yourself): target specific large or fragmented manifests instead of rewriting all; avoid Spark executor memory pressure by disabling aggressive caching during the operation (client-dependent).
Summary recommendations
-
On managed Iceberg: rely on UC’s automated maintenance; override manually only for special cases or compliance windows.
-
On external/foreign Iceberg:
- Use expireSnapshots regularly (based on business SLAs),
- Avoid deleteOrphanFiles if any other table/format could still reference the same files (including Delta),
- Run rewriteManifests periodically to keep planning efficient, especially for streaming/high-churn tables.
Cheers, Louis.