CPU usage and idle time metrics from system tables

santhiya
Databricks Partner

‎04-23-2025 04:13 AM

I need to get my compute metric, not from the UI...the system tables has not much informations, node_timeline has per minute record metric so it's difficult to calculate each compute CPU usage per day. Any way we can get the CPU usage,CPU idle time,Memory Usage for each clusters per day? I need a sql query for getting this details

0 Kudos

Louis_Frolio
Databricks Employee
Databricks Employee

‎04-23-2025 06:39 AM

To calculate CPU usage, CPU idle time, and memory usage per cluster per day, you can use the system.compute.node_timeline system table. However, since the data in this table is recorded at per-minute granularity, it’s necessary to aggregate the data to a daily level.
Below is a SQL query that you can use to calculate daily averages for CPU usage, CPU idle time, and memory utilization for each cluster:
WITH per_cluster_daily AS (
  SELECT
    cluster_id,
    DATE_TRUNC('DAY', start_time) AS day,
    AVG(cpu_user_percent + cpu_system_percent) AS avg_cpu_usage_percent, -- Average CPU usage as the sum of user and system CPU
    AVG(cpu_idle_percent) AS avg_cpu_idle_percent,                      -- Average CPU idle time percentage
    AVG(mem_used_percent) AS avg_memory_usage_percent                   -- Average memory usage percentage
  FROM
    system.compute.node_timeline
  WHERE
    start_time >= CURRENT_DATE - INTERVAL 30 DAYS -- Limit data to the last 30 days (optional)
  GROUP BY
    cluster_id,
    DATE_TRUNC('DAY', start_time)
)
SELECT
  cluster_id,
  day,
  avg_cpu_usage_percent,
  avg_cpu_idle_percent,
  avg_memory_usage_percent
FROM
  per_cluster_daily
ORDER BY
  cluster_id,
  day;
 
Note: - This query uses columns like cpu_user_percent, cpu_system_percent, cpu_idle_percent, and mem_used_percent directly from the system.compute.node_timeline table, as these metrics are captured at per-minute granularity.
0 Kudos

mas10
New Contributor II
In response to Louis_Frolio

‎09-05-2025 10:13 AM

I wanna list all the users who has access to the specific cluster and its usage like start and end time of cluster, total cost and so on do you write a query on it

0 Kudos