Hubert-Dudek
Databricks MVP

OOM error is quite common, it means that probably your partitions are too large to fit into memory. Please analyze your SPARK UI - look for data spills, and skews and try to use smaller partitions or/and more shuffle partitions.


My blog: https://databrickster.medium.com/

View solution in original post