cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Do ganglia report incorrect memory stats?

User16869510359
Esteemed Contributor

I am looking at the memory utilization of the executors and I see the heap utilization of the executor is far less than what is reported in the Ganglia. Why do ganglia report incorrect memory details.

1 ACCEPTED SOLUTION

Accepted Solutions

User16869510359
Esteemed Contributor

Ganglia reports the memory utilization at the system level.

Say for example if the JVM has Xmx value of 100 GB. At some point, it will occupy 100GB and then with a Garbage collection, it will clear off the heap. Once the GC frees up the memory, the memory is not given back to the customer. That means at the system level the JVM process is still utilizing 100GB. Although the heap usage is much less than that.

View solution in original post

1 REPLY 1

User16869510359
Esteemed Contributor

Ganglia reports the memory utilization at the system level.

Say for example if the JVM has Xmx value of 100 GB. At some point, it will occupy 100GB and then with a Garbage collection, it will clear off the heap. Once the GC frees up the memory, the memory is not given back to the customer. That means at the system level the JVM process is still utilizing 100GB. Although the heap usage is much less than that.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.