Get Level Cluster Metrics

[erwingm10]

Im looking for a way to Optimize the consumption of the jobs in my company and the last piece of data to achieve this is the statistics of the Cluster Level Metrics called Active Tasks over time. 

Do we have any way to get this? Seems easy when is already there. Since the webpage already send a JSON with that info to the UI via /graphql/getClusterLevelMetrics.

Is the a way to have this trough the Databricks REST API or there other way around to get this?

[szymon_dybczak]

Hi @erwingm10 ,

Unfortunately, currently that there is no direct endpoint in REST API to get cluster metrics. You can extract some ganglia metrics through custom scripting, but they're not so detailed like the one you looking for.

Look at below links how to implement one:

https://medium.com/@alexandr.volok/databricks-observability-collecting-ganglia-metrics-d2848d5944ee

https://alexvolok.com/2023/collecting-databricks-clusters-metrics/

But support for Cluster Level Metrics API is planned feature according to below thread, so in near future we can expect such an option:

https://community.databricks.com/t5/get-started-discussions/is-it-possible-to-view-databricks-cluste...