Deriving a relation between spark job and underlying code

Subhrajyoti
Databricks Partner

For one of our requirement, we need to derive a relation between spark job, stage ,task id with the underlying code executed after a workflow job is getting triggered using a job cluster. So far we are able to develop a relation between the Workflow job id and spark job,task,stage id. 

Please suggest how should we proceed to derive a tabular relation between the spark job id along with the underlying code in the notebook.

 

 

Regards,
Subhrajyoti Chatterjee
Technology Specialist at Azure Data Engineering Community,
Guild & Community (GnC), Analytics & AI
Cognizant Technology Solutions
Kolkata, India