topic Re: ganglia is unavailable on DBR 13.0 in Data Engineering

ganglia is unavailable on DBR 13.0

HamidHamid_Mora — Fri, 14 Apr 2023 07:53:03 GMT

We created a library in databricks to ingest ganglia metrics for all jobs in our delta tables;However end point 8652 is no more available on DBR 13.0

is there any other endpoint available ? since we need to log all metrics for all executed jobs not only for running clusters which is only visuel

or is there qny other way that I am not aware of

Thanks

Re: ganglia is unavailable on DBR 13.0

Hubert-Dudek — Fri, 14 Apr 2023 09:18:27 GMT

Ganglia is only supported on Databricks Runtime versions 12 and below. From Databricks Runtime 13, Ganglia is replaced by a new Databricks metrics system offering more features and integrations. To export metrics to external services, you can use Datadog or Azure Log Analytics integration as recommended best practices https://learn.microsoft.com/en-us/azure/databricks/clusters/clusters-manage

Re: ganglia is unavailable on DBR 13.0

HamidHamid_Mora — Fri, 14 Apr 2023 09:29:22 GMT

Thanks for your reply, however I dont see cluster metrics like cpu,memory and network in log analytics options on databricks

Re: ganglia is unavailable on DBR 13.0

rahuja — Thu, 07 Nov 2024 13:17:24 GMT

@Hubert-Dudek and @HamidHamid_Mora Is there any alternative to this? I can not see the CPU and memory metrics of the cluster even when I export the logs. Has someone sucessfully done this? How can I filter the logs containing metric once imported into ELK stack via Event-hub?

Re: ganglia is unavailable on DBR 13.0

h_h_ak — Thu, 07 Nov 2024 21:46:46 GMT

You should have a look here:

https://community.databricks.com/t5/data-engineering/azure-databricks-metrics-to-prometheus/td-p/71569