https://community.databricks.com/t5/data-engineering/i-know-my-partitions-are-skewed-is-there-anything-i-can-do-to/m-p/23496#M16221 <P>For scenarios like this, it is recommend to use a cluster with Databricks Runtime 7.3 LTS or above where AQE is enabled. </P><P></P><P>AQE dynamically handles skew in sort merge join and shuffle hash join by splitting (and replicating if needed) skewed tasks into roughly evenly sized tasks.</P><P></P> Fri, 18 Jun 2021 06:16:06 GMT sajith_appukutt 2021-06-18T06:16:06Z https://community.databricks.com/t5/data-engineering/i-know-my-partitions-are-skewed-is-there-anything-i-can-do-to/m-p/23495#M16220 <P>I know the skew in my dataset has the potential to cause issues with my job performance, so just wondering if there is anything I can do to help my performance other than repartitioning the whole dataset.</P> Thu, 17 Jun 2021 03:41:10 GMT https://community.databricks.com/t5/data-engineering/i-know-my-partitions-are-skewed-is-there-anything-i-can-do-to/m-p/23495#M16220 User16826992666 2021-06-17T03:41:10Z https://community.databricks.com/t5/data-engineering/i-know-my-partitions-are-skewed-is-there-anything-i-can-do-to/m-p/23496#M16221 <P>For scenarios like this, it is recommend to use a cluster with Databricks Runtime 7.3 LTS or above where AQE is enabled. </P><P></P><P>AQE dynamically handles skew in sort merge join and shuffle hash join by splitting (and replicating if needed) skewed tasks into roughly evenly sized tasks.</P><P></P> Fri, 18 Jun 2021 06:16:06 GMT https://community.databricks.com/t5/data-engineering/i-know-my-partitions-are-skewed-is-there-anything-i-can-do-to/m-p/23496#M16221 sajith_appukutt 2021-06-18T06:16:06Z