Hi @DB1To3 ,

How are you doing today? Good question. With an all-purpose cluster, multiple runs can share the same Spark application, so the Spark UI can definitely get confusing.

One option I would suggest is using the Jobs API to list active runs and then checking the cluster information associated with each run. You can match the cluster ID with your all-purpose cluster to get the runs using that compute.

For easier debugging and isolation, job clusters are usually a better option when possible, since each job run gets its own compute and Spark context. Hope this helps.