saurabh18cs
Honored Contributor III
  • spark.executor.memory is for JVM heap memory, while spark.executor.memoryOverhead is for non-JVM memory. The off-heap memory is outside the ambit of Garbage Collection

The total off-heap memory for a Spark executor is controlled by spark.executor.memoryOverhead. The default value for this is 10% of executor memory subject to a minimum of 384MB. This means, even if the user does not explicitly set this parameter, Spark would set aside 10% of executor memory(or 384MB whichever is higher) for VM overheads.