What is overwatch ?

User16826994223
Databricks Employee
Databricks Employee

I heard Databricks recommend overwatch for monitoring Clusters, Can anybody help like what all metrics it will provide that , how it is helpful in monitoring or better than ganglia ?

alexott
Databricks Employee
Databricks Employee

Overwatch is a different kind of tool - right now it couldn't be used for real-time monitoring, like, Ganglia. Overwatch collects data from the multiple data sources (audit logs, APIs, cluster logs, etc.), process, enrich and aggregate them following traditional Bronze/Silver/Gold approach. The data that is provided by Overwatch could be used for different purposes:

  • Cost estimation - it may provide more granular analysis, like, attributing costs to specific notebooks and users, and also overcome the limits for clusters acquired from the instance pools
  • Governance and monitoring with much longer periods of time and much cheaper compared to Azure Log Analytics or other solutions
  • Identifying the problematic workloads & optimizing them
  • ...

Overwatch is available to Databricks customers as part of the Databricks Labs. Follow the documentation for installation instructions.