cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
HamidHamid_Mora
by • New Contributor II
  • 2460 Views
  • 2 replies
  • 2 kudos

ganglia is unavailable on DBR 13.0

We created a library in databricks to ingest ganglia metrics for all jobs in our delta tables;However end point 8652 is no more available on DBR 13.0is there any other endpoint available ? since we need to log all metrics for all executed jobs not on...

Data Engineering
Reply
  • 2460 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

Ganglia is only supported on Databricks Runtime versions 12 and below. From Databricks Runtime 13, Ganglia is replaced by a new Databricks metrics system offering more features and integrations. To export metrics to external services, you can use Dat...

  • 2 kudos
1 More Replies
jamesw
by • New Contributor II
  • 2241 Views
  • 2 replies
  • 1 kudos

Ganglia not working with custom container services

Setup:custom docker container starting from the "databricksruntime/gpu-conda:cuda11" base image layer10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)multi-node, p3.8xlarge GPU computeWhen I try to view Ganglia metrics I am met with "502 Bad Gatewa...

image.png image
Data Engineering
Reply
  • 2241 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @James W​ (Customer)​ , We haven’t heard from you since the last response from @Vivian Wilfred​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the community as it can be ...

  • 1 kudos
1 More Replies
Pritesh1
by • New Contributor II
  • 2936 Views
  • 3 replies
  • 0 kudos

Resolved! Ganglia UI not showing visuals

Hello, I am trying to use Metrics and Ganglia UI to monitor the state of my clusters better. But, I am seeing that the visuals are not coming up. I have tried opening on Chrome and microsoft edge, it shows same. Is there something that I need to inst...

image
Data Engineering
Reply
  • 2936 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Pritesh1
New Contributor II
  • 0 kudos

I dont exactly know what was the issue. But, it seems to be related to some kind of network security. Apparently, my IT team had set up a separate vm and making the changes for that specific vm to be able to use Ganglia from there. I end up RDP into ...

  • 0 kudos
2 More Replies
brickster_2018
by • Esteemed Contributor
  • 5046 Views
  • 1 replies
  • 0 kudos

Resolved! Do ganglia report incorrect memory stats?

I am looking at the memory utilization of the executors and I see the heap utilization of the executor is far less than what is reported in the Ganglia. Why do ganglia report incorrect memory details.

Data Engineering
Reply
  • 5046 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
brickster_2018
Esteemed Contributor
  • 0 kudos

Ganglia reports the memory utilization at the system level. Say for example if the JVM has Xmx value of 100 GB. At some point, it will occupy 100GB and then with a Garbage collection, it will clear off the heap. Once the GC frees up the memory, th...

  • 0 kudos
User16790091296
by • Contributor II
  • 737 Views
  • 0 replies
  • 5 kudos

Some Tips & Tricks for Optimizing costs and performance (Clusters and Ganglia): [Note: This list is not exhaustive] Leverage the DataFrame or Spar...

Some Tips & Tricks for Optimizing costs and performance (Clusters and Ganglia):[Note: This list is not exhaustive]Leverage the DataFrame or SparkSQL API’s first. They use the same execution process resulting in parity in performance but they also com...

Data Engineering
Reply
  • 737 Views
  • 0 replies
  • 5 kudos
Labels