cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to handle timeout exception in Error Handle Task

NhanNguyen
Contributor III

Dear team,

 

I have a workflow like this, task_a, task_b and handle_error. How I handle any timeout exception from task_a and task_b or any future task in future and log into handle error task at the end.

Best regards,

Jensen Nguyen

NhanNguyen_1-1727750059373.png

 

1 REPLY 1

VZLA
Databricks Employee
Databricks Employee

@NhanNguyen thanks for your question!


Have you maybe consider to:

  • Define a Global Error Task: Add a handle_error task in the workflow that runs conditionally on task failure.
  • Set Failure Conditions: In the UI or through JSON, configure the handle_error task with a "run if any upstream task fails" condition.
{
    "tasks": [
        {
            "task_key": "task_a",
            "libraries": [],
            "timeout_seconds": 3600
        },
        {
            "task_key": "task_b",
            "depends_on": [{"task_key": "task_a"}],
            "libraries": [],
            "timeout_seconds": 3600
        },
        {
            "task_key": "handle_error",
            "depends_on": [
                {"task_key": "task_a"},
                {"task_key": "task_b"}
            ],
            "condition_task_state": "FAILED",
            "libraries": []
        }
    ]
}
  • Implement Custom Error Logging: Write a Python script in handle_error that aggregates errors from upstream tasks for logging or notification.



Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group