cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to handle timeout exception in Error Handle Task

NhanNguyen
Contributor III

Dear team,

 

I have a workflow like this, task_a, task_b and handle_error. How I handle any timeout exception from task_a and task_b or any future task in future and log into handle error task at the end.

Best regards,

Jensen Nguyen

NhanNguyen_1-1727750059373.png

 

1 REPLY 1

VZLA
Databricks Employee
Databricks Employee

@NhanNguyen thanks for your question!


Have you maybe consider to:

  • Define a Global Error Task: Add a handle_error task in the workflow that runs conditionally on task failure.
  • Set Failure Conditions: In the UI or through JSON, configure the handle_error task with a "run if any upstream task fails" condition.
{
    "tasks": [
        {
            "task_key": "task_a",
            "libraries": [],
            "timeout_seconds": 3600
        },
        {
            "task_key": "task_b",
            "depends_on": [{"task_key": "task_a"}],
            "libraries": [],
            "timeout_seconds": 3600
        },
        {
            "task_key": "handle_error",
            "depends_on": [
                {"task_key": "task_a"},
                {"task_key": "task_b"}
            ],
            "condition_task_state": "FAILED",
            "libraries": []
        }
    ]
}
  • Implement Custom Error Logging: Write a Python script in handle_error that aggregates errors from upstream tasks for logging or notification.



Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now