ganglia is unavailable on DBR 13.0
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-14-2023 12:53 AM
We created a library in databricks to ingest ganglia metrics for all jobs in our delta tables;However end point 8652 is no more available on DBR 13.0
is there any other endpoint available ? since we need to log all metrics for all executed jobs not only for running clusters which is only visuel
or is there qny other way that I am not aware of
Thanks
- Labels:
-
Delta Tables
-
Ganglia
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-14-2023 02:18 AM
Ganglia is only supported on Databricks Runtime versions 12 and below. From Databricks Runtime 13, Ganglia is replaced by a new Databricks metrics system offering more features and integrations. To export metrics to external services, you can use Datadog or Azure Log Analytics integration as recommended best practices https://learn.microsoft.com/en-us/azure/databricks/clusters/clusters-manage
My blog: https://databrickster.medium.com/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-14-2023 02:29 AM
Thanks for your reply, however I dont see cluster metrics like cpu,memory and network in log analytics options on databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2024 05:17 AM
@Hubert-Dudek and @HamidHamid_Mora Is there any alternative to this? I can not see the CPU and memory metrics of the cluster even when I export the logs. Has someone sucessfully done this? How can I filter the logs containing metric once imported into ELK stack via Event-hub?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2024 01:45 PM - edited 11-07-2024 01:46 PM
You should have a look here:
