I know my partitions are skewed, is there anything I can do to help my performance?

User16826992666
Databricks Employee
Databricks Employee

‎06-16-2021 08:41 PM

I know the skew in my dataset has the potential to cause issues with my job performance, so just wondering if there is anything I can do to help my performance other than repartitioning the whole dataset.

Labels:
0 Kudos

sajith_appukutt
Databricks Employee
Databricks Employee

‎06-17-2021 11:16 PM

For scenarios like this, it is recommend to use a cluster with Databricks Runtime 7.3 LTS or above where AQE is enabled.

AQE dynamically handles skew in sort merge join and shuffle hash join by splitting (and replicating if needed) skewed tasks into roughly evenly sized tasks.

View solution in original post

0 Kudos