🚨 Problem Statement: In a large-scale Databricks pipeline processing billions of rows, I faced severe performance degradation during write operations. Traditional partitioning caused executor failures, memory overload, and write times exceeding 10 hours. ⚙️ Environment Overview (Common in both the