Next, use the repartition(n) to increase your dataframe to twice the number you got earlier. For example, if num_partitions was 30, then repartition(60) prior to running your query. With half the data in each Memory Partition, I'm guessing you won't...
OK, without having your code or DAG, it's a little difficult to figure this out. But here's something that should work. First, figure out who many Memory Partitions you have. Apparently, your Memory Partitions are too big for the cluster, hence th...
Other things to consider. By any chance do you have Spot instances of Workers turned on (edge case)? I've seen where this hand-cuffs AQE. If have join, do you have the smaller table as the first table in the JOIN? Are you ANALYZE TABLE which can cha...
I'm guessing you have Shuffle Write Sizes that are > 1GB. That's when things start going down the rathole with things like Spill and OOM. Here's a few questions I have for your. Is Adaptive Query Execution enabled? Also I say in your earlier scree...
I'm guessing you are running one of more Wide transformations in your query and that is causing Skewed Shuffle Partitions. Go back to Stages tab and check out 'Shuffle Write Size/Records' row.
