Overwatch is a different kind of tool - right now it couldn't be used for real-time monitoring, like, Ganglia. Overwatch collects data from the multiple data sources (audit logs, APIs, cluster logs, etc.), process, enrich and aggregate them following traditional Bronze/Silver/Gold approach. The data that is provided by Overwatch could be used for different purposes:
- Cost estimation - it may provide more granular analysis, like, attributing costs to specific notebooks and users, and also overcome the limits for clusters acquired from the instance pools
- Governance and monitoring with much longer periods of time and much cheaper compared to Azure Log Analytics or other solutions
- Identifying the problematic workloads & optimizing them
- ...
Overwatch is available to Databricks customers as part of the Databricks Labs. Follow the documentation for installation instructions.
