topic Re: Databricks Workflow duration issues in Data Engineering

Databricks Workflow duration issues

ChsAIkrishna — Wed, 13 Nov 2024 16:44:31 GMT

There is a discrepancy in the Databricks Workflow . Currently, the start and end times printed are when the workflow is triggered, not when the actual workflow execution begins. Ideally, it should log the task execution time. The issue is that the current implementation includes queuing time, which is not the intended behavior.

For example, if the threshold is set to 60 minutes, the expectation is that the task execution time should not exceed 60 minutes, but it is currently counting the time spent in the queue, which is not correct.

Any advise to fix ?

Re: Databricks Workflow duration issues

Walter_C — Wed, 13 Nov 2024 18:26:32 GMT

This is already reported on a feature request to make sure we set a difference in between the queue time and the execution time in the job UI page, this can be tracked with id DB-I-8963

Re: Databricks Workflow duration issues

ChsAIkrishna — Thu, 14 Nov 2024 11:20:29 GMT

Thank you for your reply and tracking ID