topic Re: Distinguishing Runs Related to an All-Purpose Cluster in Data Engineering

Distinguishing Runs Related to an All-Purpose Cluster

DB1To3 — Mon, 24 Aug 2026 19:55:05 GMT

Sorry for the basic question. If I am using an all-purpose clusters in Databricks, it seems to combine all my runs together and execute them within the same Apache Spark application. There is only one application shown in the Spark UI. Unrelated runs gets mixed up together in the same Spark UI. I'm using an all-purpose cluster, not a job cluster.

Normally I send the job runs to this cluster by way of the REST API (jobs api).

How can distinguish and review the list of runs that are currently associated with a given "all-purpose" cluster? I'd like to be able to at least get a simple list of active runs on a given compute. There are different runs using different compute, and I haven't found a way to filter the list of runs, based on the related compute cluster. What am I missing?

Re: Distinguishing Runs Related to an All-Purpose Cluster

Brahmareddy — Tue, 25 Aug 2026 03:15:00 GMT

Hi @DB1To3 ,

How are you doing today? Good question. With an all-purpose cluster, multiple runs can share the same Spark application, so the Spark UI can definitely get confusing.

One option I would suggest is using the Jobs API to list active runs and then checking the cluster information associated with each run. You can match the cluster ID with your all-purpose cluster to get the runs using that compute.

For easier debugging and isolation, job clusters are usually a better option when possible, since each job run gets its own compute and Spark context. Hope this helps.

Re: Distinguishing Runs Related to an All-Purpose Cluster

balajij8 — Tue, 25 Aug 2026 04:03:15 GMT

@DB1To3

All purpose cluster combines workloads into a single application to optimize resource reuse unlike job clusters, which create isolated Spark applications per execution. Because there is no separation at the Spark application level on existing compute, all stages, tasks and metrics from unrelated runs will usually appear combined in the Spark UI.

You can query the Jobs API list-runs endpoint with active_only=true and filter the results to identify and list the runs tied to a specific compute. You can retrieve all active runs and parse them in code by matching against the cluster_instance.cluster_id field as the API usually does not accept cluster_id as a direct filter parameter. Each run object contains cluster_instance.cluster_id, run_id, job_id, run_name and state.life_cycle_state giving full visibility into what is running on the cluster. You can do this query directly using REST API or Databricks CLI

You can migrate away from all-purpose clusters to dedicated job clusters for workloads where metric isolation and diagnostics is required. Defining the compute using new_cluster or job_cluster_key in the job definition ensures that each run gets an isolated Spark application with automated resource teardown upon completion and clean boundaries between runs that eliminate metric mixing entirely.

Re: Distinguishing Runs Related to an All-Purpose Cluster

Khasim_1 — Tue, 25 Aug 2026 05:12:27 GMT

If you must use an All-Purpose cluster (e.g., for cost savings or fast start times), here is how you manage the chaos:

The "Job Name" filter in the Jobs UI: Don't look at the Spark UI first. Go to the Workflows > Jobs tab. You can filter the "Runs" list by Cluster ID or Job Name. This is the single source of truth for what is running on which compute.

Use the run_name parameter: When submitting via the REST API, always pass a unique run_name. This will show up in the Jobs UI, allowing you to filter easily.

If you are inside the Spark UI and it’s a mess, look at the "Jobs" tab within the Spark UI. It will list the job descriptions. If you haven't labeled your code, it will be impossible to tell them apart. Use spark.sparkContext.setJobDescription("My_Job_Name") inside your code. This label will appear in the Spark UI, allowing you to distinguish the runs.

Re: Distinguishing Runs Related to an All-Purpose Cluster

DB1To3 — Tue, 25 Aug 2026 17:22:13 GMT

I understand about using the API or CLI. However it is very confusing that the list of "jobs" doesn't allow you to filter on a particular compute cluster.

Nor does the "all-purpose cluster" allow you to see the list of jobs running on it. In neither place does it give you a prominent visualization of this relationship, or allow us to review the list of "jobs" on the "cluster". These entities are both very important in the workspace, and it is unfortunate to rely on the CLI or API tools. I can do that, but I'm surprised to have to create that visualization for myself.

Re: Distinguishing Runs Related to an All-Purpose Cluster

DB1To3 — Tue, 25 Aug 2026 17:26:11 GMT

I will see if there is a way to pass the job name. So far it hasn't shown up in my filter, but it might be an optional parameter that I'm not sending. Thjere is nothing in the filter that corresponds to my runs.

I think I'm sending the "run name", but possibly not the job name.

Re: Distinguishing Runs Related to an All-Purpose Cluster

DB1To3 — Tue, 25 Aug 2026 17:31:45 GMT

We are hoping to use "spark connect" in the next phase of work and I've heard it isn't trivial to use that feature with "jobs clusters". I will explore both approaches, but I'm guessing that databricks had invested more energy into making that feature compatible with the "all-purpose clusters". So that is my starting point right now.

Where jobs clusters are concerned, I don't have high expectations. I had asked in the past and was told that "spark connect" was primarily designed to be used on all-purpose clusters.

Re: Distinguishing Runs Related to an All-Purpose Cluster

Brahmareddy — Tue, 25 Aug 2026 19:30:17 GMT

Hi @DB1To3 , That makes sense. If Spark Connect is part of the next phase, starting with an all-purpose cluster sounds reasonable. In that case, I would focus on the Jobs API approach and map active runs back to the cluster ID. It would definitely be useful if Databricks provided a simpler cluster-level view of all active job runs directly.