I have a pyspark job reading the input data volume of just ~50-55GB Parquet data from a delta table on Databricks. Job is using n2-highmem-4 GCP VM and 1-15 worker with autoscaling on databricks. Each workerVM of type n2-highmem-4 has 32GB memory and 4 cores. Each VM has one executor. 22GB is allocated per executor. ie 22*15=330GB overall executor memory, which seems to be large enough for ~55GB input data. shuffle partition is set to 200. But Im getting OOM error.

• Input data volume : 55GB
• Number of worker : 15 n2-highmem-4 GCP VM and 1-15 worker with autoscaling
• Number of executor per worker : 1
• number of core per executor (or worker) : 4 ie. only 4 tasks can run in parallel
• shuffle partitions : 200
• so number of partitions per worker : 200/15 = ~13 partitions
• data per partition : 55GB/200 = ~275MB (this is just for calculation, there would be skew, some partitions will have much more data, is there a way to figure out from spark UI?)
• Overall executor memory : 22*15=330GB 
  • Spark memory (storage+execution) per worker = 0.6*(22000MB-300MB) = ~13GB

Could you please help understand why this is not sufficient leading to oom? Also is it necessary for all ~13 partitions assigned to an executor to fit in memory at once or since only 4 tasks run in parallel per executor, is it sufficient for memory to accommodate just 4 partitions at a time?