ganglia is unavailable on DBR 13.0

[HamidHamid_Mora]

We created a library in databricks to ingest ganglia metrics for all jobs in our delta tables;However end point 8652 is no more available on DBR 13.0

is there any other endpoint available ? since we need to log all metrics for all executed jobs not only for running clusters which is only visuel

or is there qny other way that I am not aware of

Thanks

[Hubert-Dudek]

Ganglia is only supported on Databricks Runtime versions 12 and below. From Databricks Runtime 13, Ganglia is replaced by a new Databricks metrics system offering more features and integrations. To export metrics to external services, you can use Datadog or Azure Log Analytics integration as recommended best practices https://learn.microsoft.com/en-us/azure/databricks/clusters/clusters-manage

[HamidHamid_Mora]

Thanks for your reply, however I dont see cluster metrics like cpu,memory and network in log analytics options on databricks

[rahuja]

@Hubert-Dudek  and @HamidHamid_Mora  Is there any alternative to this? I can not see the CPU and memory metrics of the cluster even when I export the logs. Has someone sucessfully done this? How can I filter the logs containing metric once imported into ELK stack via Event-hub?

[h_h_ak]

You should have a look here: 

https://community.databricks.com/t5/data-engineering/azure-databricks-metrics-to-prometheus/td-p/715...