Get Level Cluster Metrics
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-16-2024 08:25 AM
Im looking for a way to Optimize the consumption of the jobs in my company and the last piece of data to achieve this is the statistics of the Cluster Level Metrics called Active Tasks over time.
Do we have any way to get this? Seems easy when is already there. Since the webpage already send a JSON with that info to the UI via /graphql/getClusterLevelMetrics.
Is the a way to have this trough the Databricks REST API or there other way around to get this?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-16-2024 08:42 AM - edited 07-16-2024 08:51 AM
Hi @erwingm10 ,
Unfortunately, currently that there is no direct endpoint in REST API to get cluster metrics. You can extract some ganglia metrics through custom scripting, but they're not so detailed like the one you looking for.
Look at below links how to implement one:
https://medium.com/@alexandr.volok/databricks-observability-collecting-ganglia-metrics-d2848d5944ee
https://alexvolok.com/2023/collecting-databricks-clusters-metrics/
But support for Cluster Level Metrics API is planned feature according to below thread, so in near future we can expect such an option:
