Delta Lake 4.0 is the next major open-source release aligned with Spark 4.x, adding first-class Variant for semi-structured data, safer Type Widening, improved DROP FEATURE, better transaction log handling, and a new multi-engine story via Delta Kern...
Problem StatementIn one of my recent projects, I faced a significant challenge: Writing a huge dataset of 11,582,763,212 rows and 2,068 columns to a Databricks managed Delta table.The initial write operation took 22.4 hours using the following setup:...
Hey @Louis_Frolio ,Thank you for the thoughtful feedback and great suggestions!A few clarifications:AQE is already enabled in my setup, and it definitely helped reduce shuffle overhead during the write.Regarding Column Pruning, in this case, the fina...
