I am currently working on optimizing several Spark pipelines and wanted to gather community insights on advanced performance tuning. Typically, my workflow for traditional SQL optimization involves a deep dive into the execution plan to identify bottlenecks and develop a targeted optimization strategy. Within Spark, I’ve focused heavily on pipeline optimization, specifically addressing join performance through broadcast joins and mitigating shuffle delays.

I am interested in moving beyond these standard techniques. Specifically, I’ve been hearing from others developers about the Spark Catalyst Optimizer and the Tungsten Engine. I’m curious if anyone has practical experience or "under-the-hood" tips on how to better leverage these engines—or if there are other architectural optimizations (like Adaptive Query Execution) that you’ve found effective for high-volume ETL.