I found a scenario where the workflow UI seems to not list all the jobs that ran.
The objective is to keep trying to run a job, using the job rest api, when the service returns a status of "skipped" while another run is currently running, until the job finally starts when the other running job completes.
It seems to work fine except for one thing.
The workflows UI does not show the sub-sequent run id that successfully ran after the previous job completed.
We know that the subsequent job did run because the job notebook prints its run id on std out. And this is the id that was returned by the successful job run.
But this successful run id never shows up on the workflow UI.
Only the previous skipped run id is listed. The first job that was running can also be seen.
When I open the skipped run's driver log, the output also says' it was skipped, but when i open the std out, I can see the subsequent run's id, the one that followed the skipped run and actually ran.
So in the end, all I see on the workflow's run UI is the first job that ran and the skipped run, but I don't see the run id of the second successful run.
This job is using my general purpose cluster which is left running the whole time. I am using the 2.0 job rest api.
Anyone see anything similar?
