Databricks Ganglia

[Tahseen0354]

Hi, is there any way to get alert automatically from databricks ganglia ? That means that a developer don’t need to review the logs manually but would get notification that resources are underutilized for example.

[Hubert-Dudek]

Databricks implementation of ganglia is so limited that it makes me laughing (png snapshots and not working tabs) so I think no 🙂

You can go with datadog installation via global init script and get stats in datadog accounts.

[Tahseen0354]

Thank you for your reply.

[Kaniz · Accepted solution]

Hi @Md Tahseen Anam​ , You can install Datadog agents on cluster nodes to send Datadog metrics to your Datadog account. The following notebook demonstrates how to install a Datadog agent on a cluster using a cluster-scoped init script.

To install the Datadog agent on all clusters, use a global init script after testing the cluster-scoped init script.

Init Script: Install Datadog Agent for Spark and System Monitoring

This init script installs the Datadog agent to collect system metrics on every node in a cluster. It also configures the cluster for Spark monitoring.

Configure <init-script-folder> with the location to put the init script.
Replace <your-api-key> in the DD_API_KEY parameter with your Datadog account key.
Run this notebook to create the script datadog-install-driver-only.sh.
Configure a cluster with the datadog-install-driver-only.sh cluster-scoped init script using the UI, Databricks CLI, or by invoking the Clusters API.

Source

[Tahseen0354]

Thank you for your reply.