The off-heap memory is managed outside the executor JVM. Spark has native support to use off-heap memory. The off-heap memory is managed by Spark and not controlled by the executor JVM. Hence GC cycles on the executor do not clean up off-heap. Databricks Photon leverages the concept of off-heap widely
Below are the instances in AWS on which Off-heap is enabled by default:
c5d.18xlarge
c5d.9xlarge
i3.16xlarge
i3en.12xlarge
i3en.24xlarge
i3en.2xlarge
i3en.3xlarge
i3en.6xlarge
i3en.large
i3en.xlarge
m4.16xlarge
m5.24xlarge
m5a.12xlarge
m5a.16xlarge
m5a.24xlarge
m5a.8xlarge
m5d.12xlarge
m5d.24xlarge
m5d.4xlarge
r4.16xlarge
r5.12xlarge
r5.16xlarge
r5.24xlarge
r5.2xlarge
r5.4xlarge
r5.8xlarge
r5a.12xlarge
r5a.16xlarge
r5a.24xlarge
r5a.2xlarge
r5a.4xlarge
r5a.8xlarge
r5d.12xlarge
r5d.24xlarge
r5d.2xlarge
r5d.4xlarge
z1d.2xlarge
z1d.3xlarge
z1d.6xlarge
z1d.6xlarge
