https://community.databricks.com/t5/data-engineering/what-is-overwatch/m-p/22831#M15711 <P>Overwatch is a different kind of tool - <I>right now</I> it couldn't be used for real-time monitoring, like, Ganglia. Overwatch collects data from the multiple data sources (audit logs, APIs, cluster logs, etc.), process, enrich and aggregate them following traditional Bronze/Silver/Gold approach. The data that is provided by Overwatch could be used for different purposes:</P><UL><LI>Cost estimation - it may provide more granular analysis, like, attributing costs to specific notebooks and users, and also overcome the limits for clusters acquired from the instance pools</LI><LI>Governance and monitoring with much longer periods of time and much cheaper compared to Azure Log Analytics or other solutions</LI><LI>Identifying the problematic workloads &amp; optimizing them</LI><LI>...</LI></UL><P></P><P><A href="https://github.com/databrickslabs/overwatch" alt="https://github.com/databrickslabs/overwatch" target="_blank">Overwatch</A> is available to Databricks customers as part of the Databricks Labs. Follow the <A href="https://databrickslabs.github.io/overwatch/" alt="https://databrickslabs.github.io/overwatch/" target="_blank">documentation</A> for installation instructions.</P> Fri, 18 Jun 2021 11:33:16 GMT alexott 2021-06-18T11:33:16Z https://community.databricks.com/t5/data-engineering/what-is-overwatch/m-p/22830#M15710 <P>I heard Databricks recommend overwatch for monitoring Clusters, Can anybody help like what all metrics it will provide that , how it is helpful in monitoring or better than ganglia ?</P> Fri, 18 Jun 2021 10:57:19 GMT https://community.databricks.com/t5/data-engineering/what-is-overwatch/m-p/22830#M15710 User16826994223 2021-06-18T10:57:19Z https://community.databricks.com/t5/data-engineering/what-is-overwatch/m-p/22831#M15711 <P>Overwatch is a different kind of tool - <I>right now</I> it couldn't be used for real-time monitoring, like, Ganglia. Overwatch collects data from the multiple data sources (audit logs, APIs, cluster logs, etc.), process, enrich and aggregate them following traditional Bronze/Silver/Gold approach. The data that is provided by Overwatch could be used for different purposes:</P><UL><LI>Cost estimation - it may provide more granular analysis, like, attributing costs to specific notebooks and users, and also overcome the limits for clusters acquired from the instance pools</LI><LI>Governance and monitoring with much longer periods of time and much cheaper compared to Azure Log Analytics or other solutions</LI><LI>Identifying the problematic workloads &amp; optimizing them</LI><LI>...</LI></UL><P></P><P><A href="https://github.com/databrickslabs/overwatch" alt="https://github.com/databrickslabs/overwatch" target="_blank">Overwatch</A> is available to Databricks customers as part of the Databricks Labs. Follow the <A href="https://databrickslabs.github.io/overwatch/" alt="https://databrickslabs.github.io/overwatch/" target="_blank">documentation</A> for installation instructions.</P> Fri, 18 Jun 2021 11:33:16 GMT https://community.databricks.com/t5/data-engineering/what-is-overwatch/m-p/22831#M15711 alexott 2021-06-18T11:33:16Z