cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Do ganglia report incorrect memory stats?

brickster_2018
Databricks Employee
Databricks Employee

I am looking at the memory utilization of the executors and I see the heap utilization of the executor is far less than what is reported in the Ganglia. Why do ganglia report incorrect memory details.

1 ACCEPTED SOLUTION

Accepted Solutions

brickster_2018
Databricks Employee
Databricks Employee

Ganglia reports the memory utilization at the system level.

Say for example if the JVM has Xmx value of 100 GB. At some point, it will occupy 100GB and then with a Garbage collection, it will clear off the heap. Once the GC frees up the memory, the memory is not given back to the customer. That means at the system level the JVM process is still utilizing 100GB. Although the heap usage is much less than that.

View solution in original post

1 REPLY 1

brickster_2018
Databricks Employee
Databricks Employee

Ganglia reports the memory utilization at the system level.

Say for example if the JVM has Xmx value of 100 GB. At some point, it will occupy 100GB and then with a Garbage collection, it will clear off the heap. Once the GC frees up the memory, the memory is not given back to the customer. That means at the system level the JVM process is still utilizing 100GB. Although the heap usage is much less than that.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group