For one of our requirement, we need to derive a relation between spark job, stage ,task id with the underlying code executed after a workflow job is getting triggered using a job cluster. So far we are able to develop a relation between the Workflow job id and spark job,task,stage id.
Please suggest how should we proceed to derive a tabular relation between the spark job id along with the underlying code in the notebook.
