Get Level Cluster Metrics

erwingm10 — Tue, 16 Jul 2024 15:25:26 GMT

Im looking for a way to Optimize the consumption of the jobs in my company and the last piece of data to achieve this is the statistics of the Cluster Level Metrics called Active Tasks over time.

Do we have any way to get this? Seems easy when is already there. Since the webpage already send a JSON with that info to the UI via /graphql/getClusterLevelMetrics.

Is the a way to have this trough the Databricks REST API or there other way around to get this?

Re: Get Level Cluster Metrics

szymon_dybczak — Tue, 16 Jul 2024 15:51:09 GMT

Hi @erwingm10 ,

Unfortunately, currently that there is no direct endpoint in REST API to get cluster metrics. You can extract some ganglia metrics through custom scripting, but they're not so detailed like the one you looking for.

Look at below links how to implement one:

https://medium.com/@alexandr.volok/databricks-observability-collecting-ganglia-metrics-d2848d5944ee

https://alexvolok.com/2023/collecting-databricks-clusters-metrics/

But support for Cluster Level Metrics API is planned feature according to below thread, so in near future we can expect such an option:

https://community.databricks.com/t5/get-started-discussions/is-it-possible-to-view-databricks-cluster-metrics-using-rest-api/td-p/57202

topic Get Level Cluster Metrics in Data Engineering

Get Level Cluster Metrics

Re: Get Level Cluster Metrics