topic Cluster Health Dashboard in Data Engineering

Cluster Health Dashboard

brickster_2018 — Fri, 25 Jun 2021 16:57:44 GMT

Is there a cluster health dashboard which has the details of the total number of running interactive cluster, the total number of job clusters? Also Flag clusters with issues.

Re: Cluster Health Dashboard

brickster_2018 — Fri, 25 Jun 2021 16:58:01 GMT

Right now there is no inbuilt dashboard giving this information. However, it's easy to develop one using the Databricks REST API for clusters.

List API will list the clusters and based on the STATE field we can determine if the cluster is up and running or not

Cluster events API gives the details of the events on the cluster. This is useful to see if there are any warnings or issues on the cluster. eg: Driver unresponsive

https://docs.databricks.com/dev-tools/api/latest/clusters.html#list

Re: Cluster Health Dashboard

valeryuaba — Mon, 17 Jul 2023 09:06:26 GMT

Thanks!