topic Re: Is there any way to monitor the CPU, disk and memory usage of a cluster while a job is running? in Data Engineering

Is there any way to monitor the CPU, disk and memory usage of a cluster while a job is running?

SaravananPalani — Thu, 23 Aug 2018 11:08:35 GMT

I am looking for something preferably similar to Windows task manager which we can use for monitoring the CPU, memory and disk usage for local desktop.

Re: Is there any way to monitor the CPU, disk and memory usage of a cluster while a job is running?

ThomasKastl — Wed, 13 Feb 2019 17:17:16 GMT

I would also find this really really useful.

Re: Is there any way to monitor the CPU, disk and memory usage of a cluster while a job is running?

User16301467513 — Thu, 30 May 2019 20:11:05 GMT

Spark UI can give you access to some of this information, just not in real-time. It's also intended for Spark-specific performance information such as job and task breakdowns.

Ganglia metrics can give you real-time metrics along these lines both in real-time and historically.

In the Clusters page for your particular cluster, select the "Metrics" link and you'll have access to the "Ganglia UI" link (for real-time) and the historical snapshots list.screen-shot-2019-05-30-at-40457-pm.png

You can find out more at the Metrics documentation page:

https://docs.databricks.com/user-guide/clusters/metrics.html

Re: Is there any way to monitor the CPU, disk and memory usage of a cluster while a job is running?

Anonymous — Wed, 07 Oct 2020 18:41:53 GMT

Ganglia metric are not that much helpful and also with cluster start you lose old data .

Question is how to get live metrics and view historical data .

OMS agent are best in that case. i used in Azure databricks and its wonderful .

should be doable in AWS as well with some modification.

Re: Is there any way to monitor the CPU, disk and memory usage of a cluster while a job is running?

Pelicanine — Tue, 13 Oct 2020 10:17:15 GMT

Ganglia metrics can give you real-time metrics along these lines both in real-time and historically. mcdvoice

Re: Is there any way to monitor the CPU, disk and memory usage of a cluster while a job is running?

youssefmrini — Wed, 01 Feb 2023 10:10:18 GMT

You can use the Ganglia UI to track the CPU, Network, Disk, and Memory. Keep in mind that Ganglia UI in a snapshot displayed every 15 minutes

Re: Is there any way to monitor the CPU, disk and memory usage of a cluster while a job is running?

Rajeev_Basu — Fri, 03 Feb 2023 11:35:37 GMT

as mentioned by few - Ganglia UI can be used to track it. we use the same in our projects.

Re: Is there any way to monitor the CPU, disk and memory usage of a cluster while a job is running?

Meghala — Sat, 04 Feb 2023 14:07:17 GMT

Which is real real time matrics

Re: Is there any way to monitor the CPU, disk and memory usage of a cluster while a job is running?

hitech88 — Sat, 04 Feb 2023 19:57:28 GMT

Some important info to look in Gangalia UI in CPU, memory and server load charts to spot the problem:

CPU chart :

User %
Idle %

High percentage of user % indicates heavy CPU usage in the cluster.

Memory chart :

Use %
Free %
Swap %

If you see purple line over red line in memory chart then it indicates memory swapping and also highlighting high memory usage.

Server Load Distribution Chart:

Absence of red squares indicates balanced load on the cluster. Presence of red squares means there is hot spot where load is more.