gchandra
Databricks Employee
Databricks Employee

Try setting  spark.sql.shuffle.partitions to auto



~