How to handle timeout exception in Error Handle Task

NhanNguyen — Tue, 01 Oct 2024 02:35:32 GMT

Dear team,

I have a workflow like this, task_a, task_b and handle_error. How I handle any timeout exception from task_a and task_b or any future task in future and log into handle error task at the end.

Best regards,

Jensen Nguyen

Re: How to handle timeout exception in Error Handle Task

VZLA — Wed, 11 Dec 2024 19:05:11 GMT

@NhanNguyen thanks for your question!

Have you maybe consider to:

Define a Global Error Task: Add a handle_error task in the workflow that runs conditionally on task failure.
Set Failure Conditions: In the UI or through JSON, configure the handle_error task with a "run if any upstream task fails" condition.

{ "tasks": [ { "task_key": "task_a", "libraries": [], "timeout_seconds": 3600 }, { "task_key": "task_b", "depends_on": [{"task_key": "task_a"}], "libraries": [], "timeout_seconds": 3600 }, { "task_key": "handle_error", "depends_on": [ {"task_key": "task_a"}, {"task_key": "task_b"} ], "condition_task_state": "FAILED", "libraries": [] } ] }

Implement Custom Error Logging: Write a Python script in handle_error that aggregates errors from upstream tasks for logging or notification.

topic How to handle timeout exception in Error Handle Task in Data Engineering

How to handle timeout exception in Error Handle Task

Re: How to handle timeout exception in Error Handle Task