This is very General question more on the Design Side on Observability.

There are 500+ Data Pipeline build in healthcare domain using Azure and AWS Databricks.

May someone please help me how to design a system :-

1. Continuous track system health and behavior using telemetry data across multiple platform.

2. Determine the Issue Root cause and Scope

3. Identified Anamolies failure

I got something from chat-gpt, but if anyone has implemented similar solution in Production, Highly appreciate if they can share their learning, like [They tech stack used with very very high level on use of each component.

Thank You So Much.