Comment
06-26-2025
07:15 AM
06-26-2025
07:15 AM
My Tips:
- Disable Adaptive Query Execution (AQE).
- Significantly reduce spark.sql.shuffle.partitions (set it between 1 and the total number of available cores).
Since unit tests typically process a small amount of data, using the default values for these settings can negatively impact test performance.
Example: Simply reducing the number of shuffle partitions to 1 cut our unit test execution time from 14 minutes to just 7 minutes.