There is a discrepancy in the Databricks Workflow . Currently, the start and end times printed are when the workflow is triggered, not when the actual workflow execution begins. Ideally, it should log the task execution time. The issue is that the current implementation includes queuing time, which is not the intended behavior.
For example, if the threshold is set to 60 minutes, the expectation is that the task execution time should not exceed 60 minutes, but it is currently counting the time spent in the queue, which is not correct.
Any advise to fix ?
