Hi @minhhung0507 

How are you doing today?, as per understanding, It looks like some of your pipelines are getting stuck without failing, making it hard to detect them in time. A good way to handle this is by setting a timeout for each pipeline so that if it runs too long, it gets restarted automatically. You can also add a heartbeat check, where each pipeline updates a log or database with a timestamp while it’s running—if a pipeline stops updating, you know it’s stuck and can trigger an alert or restart it. Using Databricks monitoring tools or cloud services like AWS CloudWatch or Azure Monitor can help track running jobs and send alerts if something is taking longer than expected. Also, make sure your thread pool isn’t overloaded, as too many pipelines running at once might be causing some to freeze. These steps should help you catch and fix stuck pipelines before they cause data loss.

Regards,

Brahma