- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-12-2025 12:08 PM
HI @JameDavi_51481 , Hope you tried this approach for enabling iceberg metadata along with delta format :
ALTER TABLE internal_poc_iceberg.iceberg_poc.clickstream_gold_sink_dlt
SET TBLPROPERTIES (
'delta.columnMapping.mode' = 'name',
'delta.enableIcebergCompatV2' = 'true',
'delta.universalFormat.enabledFormats' = 'iceberg'
);
Please let me know otherwise. If you used it and still looking for fast reorg with complete rewrite you can tweak cluster settings or configuration to make it faster.
1. Use a cluster with high parallelism
Use a larger cluster (more worker nodes) with:
High I/O throughput (EBS-optimized in AWS, or Premium SSD in Azure)
High memory-to-core ratio (e.g., i3, r5d, m5d instances in AWS)
Try using photon-enabled clusters if available — Photon often improves performance of IO-heavy workloads.
2. Run upgrades in parallel
If you're upgrading multiple tables, batch them in parallel using job clusters or workflows.