topic Re: What is off-heap memory? For which all instances off-heap is enabled by default? in Data Engineering https://community.databricks.com/t5/data-engineering/what-is-off-heap-memory-for-which-all-instances-off-heap-is/m-p/19810#M13339 <P>The off-heap memory is managed outside the executor JVM. Spark has native support to use off-heap memory. The off-heap memory is managed by Spark and not controlled by the executor JVM. Hence GC cycles on the executor do not clean up off-heap. Databricks Photon leverages the concept of off-heap widely</P><P></P><P>Below are the instances in AWS on which Off-heap is enabled by default:</P><P></P><P>c5d.18xlarge</P><P>c5d.9xlarge</P><P>i3.16xlarge</P><P>i3en.12xlarge</P><P>i3en.24xlarge</P><P>i3en.2xlarge</P><P>i3en.3xlarge</P><P>i3en.6xlarge</P><P>i3en.large</P><P>i3en.xlarge</P><P>m4.16xlarge</P><P>m5.24xlarge</P><P>m5a.12xlarge</P><P>m5a.16xlarge</P><P>m5a.24xlarge</P><P>m5a.8xlarge</P><P>m5d.12xlarge</P><P>m5d.24xlarge</P><P>m5d.4xlarge</P><P>r4.16xlarge</P><P>r5.12xlarge</P><P>r5.16xlarge</P><P>r5.24xlarge</P><P>r5.2xlarge</P><P>r5.4xlarge</P><P>r5.8xlarge</P><P>r5a.12xlarge</P><P>r5a.16xlarge</P><P>r5a.24xlarge</P><P>r5a.2xlarge</P><P>r5a.4xlarge</P><P>r5a.8xlarge</P><P>r5d.12xlarge</P><P>r5d.24xlarge</P><P>r5d.2xlarge</P><P>r5d.4xlarge</P><P>z1d.2xlarge</P><P>z1d.3xlarge</P><P>z1d.6xlarge</P><P>z1d.6xlarge</P> Fri, 25 Jun 2021 13:55:42 GMT brickster_2018 2021-06-25T13:55:42Z What is off-heap memory? For which all instances off-heap is enabled by default? https://community.databricks.com/t5/data-engineering/what-is-off-heap-memory-for-which-all-instances-off-heap-is/m-p/19809#M13338 Fri, 25 Jun 2021 13:55:16 GMT https://community.databricks.com/t5/data-engineering/what-is-off-heap-memory-for-which-all-instances-off-heap-is/m-p/19809#M13338 brickster_2018 2021-06-25T13:55:16Z Re: What is off-heap memory? For which all instances off-heap is enabled by default? https://community.databricks.com/t5/data-engineering/what-is-off-heap-memory-for-which-all-instances-off-heap-is/m-p/19810#M13339 <P>The off-heap memory is managed outside the executor JVM. Spark has native support to use off-heap memory. The off-heap memory is managed by Spark and not controlled by the executor JVM. Hence GC cycles on the executor do not clean up off-heap. Databricks Photon leverages the concept of off-heap widely</P><P></P><P>Below are the instances in AWS on which Off-heap is enabled by default:</P><P></P><P>c5d.18xlarge</P><P>c5d.9xlarge</P><P>i3.16xlarge</P><P>i3en.12xlarge</P><P>i3en.24xlarge</P><P>i3en.2xlarge</P><P>i3en.3xlarge</P><P>i3en.6xlarge</P><P>i3en.large</P><P>i3en.xlarge</P><P>m4.16xlarge</P><P>m5.24xlarge</P><P>m5a.12xlarge</P><P>m5a.16xlarge</P><P>m5a.24xlarge</P><P>m5a.8xlarge</P><P>m5d.12xlarge</P><P>m5d.24xlarge</P><P>m5d.4xlarge</P><P>r4.16xlarge</P><P>r5.12xlarge</P><P>r5.16xlarge</P><P>r5.24xlarge</P><P>r5.2xlarge</P><P>r5.4xlarge</P><P>r5.8xlarge</P><P>r5a.12xlarge</P><P>r5a.16xlarge</P><P>r5a.24xlarge</P><P>r5a.2xlarge</P><P>r5a.4xlarge</P><P>r5a.8xlarge</P><P>r5d.12xlarge</P><P>r5d.24xlarge</P><P>r5d.2xlarge</P><P>r5d.4xlarge</P><P>z1d.2xlarge</P><P>z1d.3xlarge</P><P>z1d.6xlarge</P><P>z1d.6xlarge</P> Fri, 25 Jun 2021 13:55:42 GMT https://community.databricks.com/t5/data-engineering/what-is-off-heap-memory-for-which-all-instances-off-heap-is/m-p/19810#M13339 brickster_2018 2021-06-25T13:55:42Z