I have a query. I have multiple job (workflow)present in my workspace. Those job runs regularly. Multiple task present in those jobs. Few task having notebook that contain for each code in it. now when a job runs that particular task execute the for each loop and create multiple iteration of that loop. now My requirement is I want to capture those iteration related information via notebook using databricks sd or rest api (which I can see in databricks workspace UI)?

Interation level information like taskid-iteration task run id,start time,end date, duration, status etc

I want to see how many iteration present in that for each loop and how many successfull and failed.

Already I can capture job-run-task level information.

suggest me some doc/code sample/explaination etc.

TIA