cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

What is off-heap memory? For which all instances off-heap is enabled by default?

brickster_2018
Esteemed Contributor
 
1 ACCEPTED SOLUTION

Accepted Solutions

brickster_2018
Esteemed Contributor

The off-heap memory is managed outside the executor JVM. Spark has native support to use off-heap memory. The off-heap memory is managed by Spark and not controlled by the executor JVM. Hence GC cycles on the executor do not clean up off-heap. Databricks Photon leverages the concept of off-heap widely

Below are the instances in AWS on which Off-heap is enabled by default:

c5d.18xlarge

c5d.9xlarge

i3.16xlarge

i3en.12xlarge

i3en.24xlarge

i3en.2xlarge

i3en.3xlarge

i3en.6xlarge

i3en.large

i3en.xlarge

m4.16xlarge

m5.24xlarge

m5a.12xlarge

m5a.16xlarge

m5a.24xlarge

m5a.8xlarge

m5d.12xlarge

m5d.24xlarge

m5d.4xlarge

r4.16xlarge

r5.12xlarge

r5.16xlarge

r5.24xlarge

r5.2xlarge

r5.4xlarge

r5.8xlarge

r5a.12xlarge

r5a.16xlarge

r5a.24xlarge

r5a.2xlarge

r5a.4xlarge

r5a.8xlarge

r5d.12xlarge

r5d.24xlarge

r5d.2xlarge

r5d.4xlarge

z1d.2xlarge

z1d.3xlarge

z1d.6xlarge

z1d.6xlarge

View solution in original post

1 REPLY 1

brickster_2018
Esteemed Contributor

The off-heap memory is managed outside the executor JVM. Spark has native support to use off-heap memory. The off-heap memory is managed by Spark and not controlled by the executor JVM. Hence GC cycles on the executor do not clean up off-heap. Databricks Photon leverages the concept of off-heap widely

Below are the instances in AWS on which Off-heap is enabled by default:

c5d.18xlarge

c5d.9xlarge

i3.16xlarge

i3en.12xlarge

i3en.24xlarge

i3en.2xlarge

i3en.3xlarge

i3en.6xlarge

i3en.large

i3en.xlarge

m4.16xlarge

m5.24xlarge

m5a.12xlarge

m5a.16xlarge

m5a.24xlarge

m5a.8xlarge

m5d.12xlarge

m5d.24xlarge

m5d.4xlarge

r4.16xlarge

r5.12xlarge

r5.16xlarge

r5.24xlarge

r5.2xlarge

r5.4xlarge

r5.8xlarge

r5a.12xlarge

r5a.16xlarge

r5a.24xlarge

r5a.2xlarge

r5a.4xlarge

r5a.8xlarge

r5d.12xlarge

r5d.24xlarge

r5d.2xlarge

r5d.4xlarge

z1d.2xlarge

z1d.3xlarge

z1d.6xlarge

z1d.6xlarge

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group