Deriving a relation between spark job and underlying code
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-19-2024 08:51 PM
For one of our requirement, we need to derive a relation between spark job, stage ,task id with the underlying code executed after a workflow job is getting triggered using a job cluster. So far we are able to develop a relation between the Workflow job id and spark job,task,stage id.
Please suggest how should we proceed to derive a tabular relation between the spark job id along with the underlying code in the notebook.
Regards,
Subhrajyoti Chatterjee
Technology Specialist at Azure Data Engineering Community,
Guild & Community (GnC), Analytics & AI
Cognizant Technology Solutions
Kolkata, India
Subhrajyoti Chatterjee
Technology Specialist at Azure Data Engineering Community,
Guild & Community (GnC), Analytics & AI
Cognizant Technology Solutions
Kolkata, India
Labels:
- Labels:
-
Workflows