brickster_2018
Databricks Employee
Databricks Employee

The off-heap memory is managed outside the executor JVM. Spark has native support to use off-heap memory. The off-heap memory is managed by Spark and not controlled by the executor JVM. Hence GC cycles on the executor do not clean up off-heap. Databricks Photon leverages the concept of off-heap widely

Below are the instances in AWS on which Off-heap is enabled by default:

c5d.18xlarge

c5d.9xlarge

i3.16xlarge

i3en.12xlarge

i3en.24xlarge

i3en.2xlarge

i3en.3xlarge

i3en.6xlarge

i3en.large

i3en.xlarge

m4.16xlarge

m5.24xlarge

m5a.12xlarge

m5a.16xlarge

m5a.24xlarge

m5a.8xlarge

m5d.12xlarge

m5d.24xlarge

m5d.4xlarge

r4.16xlarge

r5.12xlarge

r5.16xlarge

r5.24xlarge

r5.2xlarge

r5.4xlarge

r5.8xlarge

r5a.12xlarge

r5a.16xlarge

r5a.24xlarge

r5a.2xlarge

r5a.4xlarge

r5a.8xlarge

r5d.12xlarge

r5d.24xlarge

r5d.2xlarge

r5d.4xlarge

z1d.2xlarge

z1d.3xlarge

z1d.6xlarge

z1d.6xlarge

View solution in original post