Custom JobGroup in Spark UI for cluster with multiple executions

FerArribas — Fri, 02 Feb 2024 16:01:47 GMT

Does anyone know what the first digits of the jobgroup that are shown in the spark ui mean when using all purpose clusters to launch multiple jobs?

Right now the pattern is something like: [id_random]_job_[jod_id]_run-[run_id]_action_[action].

Re: Custom JobGroup in Spark UI for cluster with multiple executions

saikumar246 — Tue, 06 Feb 2024 05:46:02 GMT

The first digits of the jobgroup that are shown in the spark UI are execContextId and cmdId(Command_ID).

You can think of the execContextId as some kind of “REPL ID”

For example, if you take the below job group ID as an example,

jobGroupId: Option[String] = Some(3482599387390558128_6163657922384662685_job-532427916265395-run-416293225333467-action-6638250854918794)

execContextId = 3482599387390558128

cmdId = 6163657922384662685

executionId = job-532427916265395-run-416293225333467-action-6638250854918794