Comment
Databricks Partner

My Tips:

  • Disable Adaptive Query Execution (AQE).
  • Significantly reduce spark.sql.shuffle.partitions (set it between 1 and the total number of available cores).

Since unit tests typically process a small amount of data, using the default values for these settings can negatively impact test performance.

Example: Simply reducing the number of shuffle partitions to 1 cut our unit test execution time from 14 minutes to just 7 minutes.