Hi @mkwparth ,

It seems you have too small cluster for your workload. Memory Swap Utilizations measures how much memory the JVM is spilling to disk because physical RAM is exhausted. When memory pressure is high (e.g. large joins, shuffles, caching), Spark will spill shuffle data to disk and evict cache data to disk.

So to put it simple - either your workload doesn't fit your cluster which means you're processing a lot of data. Of your workload is heavy in wide transformation, i.e a lot of joins, group by.

You can try to scale out your cluster a bit. Maybe try to use F8 or F16 with at least 16-32 GB RAM.