If you must use an All-Purpose cluster (e.g., for cost savings or fast start times), here is how you manage the chaos:

The "Job Name" filter in the Jobs UI: Don't look at the Spark UI first. Go to the Workflows > Jobs tab. You can filter the "Runs" list by Cluster ID or Job Name. This is the single source of truth for what is running on which compute.

Use the run_name parameter: When submitting via the REST API, always pass a unique run_name. This will show up in the Jobs UI, allowing you to filter easily.

If you are inside the Spark UI and it’s a mess, look at the "Jobs" tab within the Spark UI. It will list the job descriptions. If you haven't labeled your code, it will be impossible to tell them apart. Use spark.sparkContext.setJobDescription("My_Job_Name") inside your code. This label will appear in the Spark UI, allowing you to distinguish the runs.