Jobs

ramsai — Wed, 04 Feb 2026 15:47:48 GMT

Is there a way to find out how many workers or cores are being utilized in a job cluster? If so, could you please explain how to check this?

Re: Jobs

saurabh18cs — Thu, 05 Feb 2026 11:10:28 GMT

Hi @ramsai

do you need this => Event log (from cluster details)

Re: Jobs

SteveOstrowski — Sun, 08 Mar 2026 02:16:54 GMT

Hi @ramsai,

Great question! There are several ways to check how many workers and cores are being utilized in a Databricks job cluster. I will walk through each option from simplest to most advanced.

OPTION 1: CLUSTER METRICS TAB (QUICKEST WAY)

While your job is running (or after it completes, for up to 30 days), you can view resource utilization directly in the UI:

1. Go to your job run and click on the task that ran.
2. Click the cluster link to open the cluster details page.
3. Select the "Metrics" tab.

Here you will see near-real-time charts (data collected every minute) including:

- CPU utilization broken down by mode (user, system, idle, iowait)
- Memory usage (used, free, buffer, cached)
- Network bytes sent and received
- Filesystem space

You can also switch to "Spark Metrics" from the dropdown to see:

- Active tasks (which tells you how many cores are actively doing work)
- Task completion and failure rates
- Shuffle read/write bytes

To drill into individual workers, use the "All nodes" dropdown to inspect each node separately. This is helpful for spotting if one worker is overloaded while others are idle.

Documentation: https://docs.databricks.com/aws/en/compute/cluster-metrics

OPTION 2: SPARK UI (EXECUTOR-LEVEL DETAIL)

From the cluster details page, click the "Spark UI" tab. This gives you the standard Apache Spark web UI where you can:

- See the Executors tab, which shows each worker (executor) and its core count, memory, active tasks, and GC time
- Check the Stages tab to see how tasks are distributed across cores
- Identify data skew or uneven core usage across workers

This is the most granular way to see exactly how many cores each worker has and whether they are all being utilized.

Documentation: https://docs.databricks.com/aws/en/compute/clusters-manage

OPTION 3: CLUSTER DETAILS PAGE (WORKER COUNT)

On the compute details page for your running cluster, Databricks shows the number of currently allocated workers. If you have autoscaling enabled, you can compare the allocated count against your configured min and max to see if the cluster has scaled up or down.

Documentation: https://docs.databricks.com/aws/en/compute/configure

OPTION 4: SYSTEM TABLES (PROGRAMMATIC / HISTORICAL ANALYSIS)

For a programmatic and historical approach, Databricks provides system tables that let you query utilization data using SQL. This is especially useful for analyzing past job runs.

Key tables:

1. system.compute.node_timeline - Minute-by-minute CPU, memory, network, and disk metrics per node. Columns include cpu_user_percent, cpu_system_percent, cpu_wait_percent, and mem_used_percent.

2. system.compute.clusters - Configuration info including worker_count, min_autoscale_workers, max_autoscale_workers, driver_node_type, and worker_node_type.

3. system.compute.node_types - Hardware specs per instance type, including core_count (number of vCPUs) and memory_mb.

Example query to find total cores and CPU utilization for a job cluster:

SELECT
c.cluster_name,
c.worker_count,
nt.core_count AS cores_per_worker,
c.worker_count * nt.core_count AS total_worker_cores,
AVG(n.cpu_user_percent + n.cpu_system_percent) AS avg_cpu_utilization
FROM system.compute.clusters c
JOIN system.compute.node_types nt
ON c.worker_node_type = nt.node_type
LEFT JOIN system.compute.node_timeline n
ON c.cluster_id = n.cluster_id
WHERE c.cluster_name = '<your_job_cluster_name>'
GROUP BY c.cluster_name, c.worker_count, nt.core_count

Note: These system tables only include records for all-purpose and job clusters (not serverless or SQL warehouses). Nodes running less than 10 minutes may not appear.

Documentation: https://docs.databricks.com/en/admin/system-tables/compute.html

QUICK SUMMARY

- For a quick visual check while a job runs: use the Metrics tab on the cluster details page.
- For detailed per-executor core usage: use the Spark UI Executors tab.
- For historical or programmatic analysis: query the system.compute tables.
- To find the core count for your instance type: query system.compute.node_types or check your cloud provider docs.

Hope this helps! Let me know if you have any follow-up questions.

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.

topic Re: Jobs in Data Engineering

Jobs

Re: Jobs

Re: Jobs