https://community.databricks.com/t5/technical-blog/overwatch-the-observability-tool-for-databricks/ba-p/54120 <H2 class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj">What is Overwatch?</H2> <P class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj" data-selectable-paragraph=""><span class="lia-inline-image-display-wrapper lia-image-align-right" image-alt="overwatch.png" style="width: 376px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5377iA6EB1B28F7C51BB7/image-dimensions/376x376?v=v2" width="376" height="376" role="button" title="overwatch.png" alt="overwatch.png" /></span>Overwatch is an observability tool which helps you to monitor spending on your clouds and track<SPAN>&nbsp;</SPAN><STRONG class="li fs">usage</STRONG><SPAN>&nbsp;</SPAN>in various dimensions. It works by collecting job and audit log data, then joining it with data from the Databricks REST API and other sources available in the platform.&nbsp; This data is processed into a <A href="https://databrickslabs.github.io/overwatch/dataengineer/#overwatch-erd" target="_self">set of tables</A> that describe the ongoing activity of your Databricks Workspace(s).&nbsp;</P> <P class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj" data-selectable-paragraph="">Overwatch is maintained as part of <A href="https://www.databricks.com/learn/labs" target="_self">Databricks Labs</A>&nbsp;and supports all the major clouds: Azure, AWS, and GCP. In this post we will look at a variety of analytics made possible by Overwatch, then discuss what a multi-Workspace deployment is and how to implement it!</P> <H3 class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj">&nbsp;</H3> <H3 class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj"><STRONG>Features of Overwatch</STRONG></H3> <P class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj" data-selectable-paragraph=""><STRONG class="li fs">Monitoring Workspaces</STRONG></P> <OL class=""> <LI id="744f" class="lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me mf mg bj" data-selectable-paragraph="">Total spend&nbsp;</LI> </OL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SriramMohanty_0-1701149209768.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5354i149BA7844ECE08A1/image-size/large?v=v2&amp;px=999" role="button" title="SriramMohanty_0-1701149209768.png" alt="SriramMohanty_0-1701149209768.png" /></span></P> <P class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj" data-selectable-paragraph="">2. Cluster count by type&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SriramMohanty_2-1701149336228.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5356i40C474DBBB1481B9/image-size/large?v=v2&amp;px=999" role="button" title="SriramMohanty_2-1701149336228.png" alt="SriramMohanty_2-1701149336228.png" /></span></P> <P class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj" data-selectable-paragraph="">3. DBU cost vs compute cost&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SriramMohanty_3-1701149813391.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5357iFAC742865D07394E/image-size/large?v=v2&amp;px=999" role="button" title="SriramMohanty_3-1701149813391.png" alt="SriramMohanty_3-1701149813391.png" /></span></P> <P>&nbsp;</P> <P class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj" data-selectable-paragraph=""><STRONG class="li fs">Monitoring Clusters</STRONG></P> <OL class=""> <LI id="93d2" class="lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me mf mg bj" data-selectable-paragraph="">Most expensive clusters by day</LI> </OL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SriramMohanty_0-1701183538445.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5362iD9646AA28E04BFC3/image-size/large?v=v2&amp;px=999" role="button" title="SriramMohanty_0-1701183538445.png" alt="SriramMohanty_0-1701183538445.png" /></span></P> <P><SPAN>2. DBU spend by cluster type</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SriramMohanty_1-1701183574382.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5363iB8B6263781E600E0/image-size/large?v=v2&amp;px=999" role="button" title="SriramMohanty_1-1701183574382.png" alt="SriramMohanty_1-1701183574382.png" /></span></P> <P><SPAN>3. Cluster node types</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SriramMohanty_2-1701183606944.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5364iD6047AD5591A1D19/image-size/medium?v=v2&amp;px=400" role="button" title="SriramMohanty_2-1701183606944.png" alt="SriramMohanty_2-1701183606944.png" /></span></P> <P><SPAN>4.Percentage of auto-scaling clusters</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SriramMohanty_3-1701183629800.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5365iB4DB7D5B5C97E100/image-size/large?v=v2&amp;px=999" role="button" title="SriramMohanty_3-1701183629800.png" alt="SriramMohanty_3-1701183629800.png" /></span></P> <P><SPAN>5. Scale up time of clusters without pools</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SriramMohanty_4-1701183781343.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5366iF06E69C00133FF92/image-size/large?v=v2&amp;px=999" role="button" title="SriramMohanty_4-1701183781343.png" alt="SriramMohanty_4-1701183781343.png" /></span></P> <P><SPAN>6.Cluster failure state and count of failures</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SriramMohanty_5-1701183812941.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5367i708DC51751C82E84/image-size/large?v=v2&amp;px=999" role="button" title="SriramMohanty_5-1701183812941.png" alt="SriramMohanty_5-1701183812941.png" /></span></P> <P class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj" data-selectable-paragraph=""><STRONG class="li fs">Monitoring Jobs</STRONG></P> <OL class=""> <LI id="13d6" class="lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me mf mg bj" data-selectable-paragraph="">DBUs by Workflow by Workspace by date</LI> </OL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SriramMohanty_6-1701184000914.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5368i703BE51A329446F9/image-size/large?v=v2&amp;px=999" role="button" title="SriramMohanty_6-1701184000914.png" alt="SriramMohanty_6-1701184000914.png" /></span></P> <P><SPAN>2. Jobs running in Interactive clusters</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SriramMohanty_7-1701184033554.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5369i71FC8A03CD38A017/image-size/large?v=v2&amp;px=999" role="button" title="SriramMohanty_7-1701184033554.png" alt="SriramMohanty_7-1701184033554.png" /></span></P> <P><SPAN>3.Daily job status distribution</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SriramMohanty_8-1701184057763.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5370i4AE6499505DA0031/image-size/large?v=v2&amp;px=999" role="button" title="SriramMohanty_8-1701184057763.png" alt="SriramMohanty_8-1701184057763.png" /></span></P> <P><SPAN>4.Impact of failure by Workspace</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SriramMohanty_9-1701184097575.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5371iE69EB3DEFA51FCA2/image-size/large?v=v2&amp;px=999" role="button" title="SriramMohanty_9-1701184097575.png" alt="SriramMohanty_9-1701184097575.png" /></span></P> <P class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj" data-selectable-paragraph="">Here are some other analyses you can perform with Overwatch:</P> <OL class=""> <LI id="f7a5" class="lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me mf mg bj" data-selectable-paragraph=""><STRONG>Last 30 days spend</STRONG>&nbsp;<BR />Aggregate cost of cluster spend in all workspaces for the last 30 days.<BR /><BR /></LI> <LI id="a06f" class="lg lh fr li b lj ng ll lm ln nh lp lq lr ni lt lu lv nj lx ly lz nk mb mc md me mf mg bj" data-selectable-paragraph=""><STRONG>Month-over-month change in spend<BR /></STRONG>Percentage change of cluster spend compared with previous month. For example, if the percentages drop below zero, it signifies that usage is down from the previous month, and vice versa.<BR /><BR /></LI> <LI id="c9d7" class="lg lh fr li b lj ng ll lm ln nh lp lq lr ni lt lu lv nj lx ly lz nk mb mc md me mf mg bj" data-selectable-paragraph=""><STRONG>Top 3 cluster spend by workspace in the last 30 days</STRONG><BR />Provides information on the top three clusters that spend the most, per Workspace.<BR /><BR /></LI> <LI id="5d3e" class="lg lh fr li b lj ng ll lm ln nh lp lq lr ni lt lu lv nj lx ly lz nk mb mc md me mf mg bj" data-selectable-paragraph=""><STRONG>Week-over-week top 10 fastest growing clusters by Workspace</STRONG> <BR />Top 10 clusters with fastest growth in spend compared with previous week. For example, if the percentages drop below zero, it signifies dip in growth previous week, and vice versa.<BR /><BR /></LI> <LI id="fdbc" class="lg lh fr li b lj ng ll lm ln nh lp lq lr ni lt lu lv nj lx ly lz nk mb mc md me mf mg bj" data-selectable-paragraph=""><STRONG>Last 7 days of spend by Databricks Workflow<BR /></STRONG>Expenses for each job in the last 7 days.<BR /><BR /></LI> <LI id="0ce8" class="lg lh fr li b lj ng ll lm ln nh lp lq lr ni lt lu lv nj lx ly lz nk mb mc md me mf mg bj" data-selectable-paragraph=""><STRONG>Last 7 days of spend for Databricks Workflows executed on interactive clusters</STRONG><BR />Expenses for jobs performed on interactive clustersin the previous 7 days</LI> </OL> <H2 class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj"><STRONG class="li fs">What is a multi-Workspace deployment of Overwatch?</STRONG></H2> <P class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj" data-selectable-paragraph=""><SPAN>If you possess multiple Databricks workspaces and wish to oversee them collectively, you can implement a multi-Workspace deployment. If the prospect of monitoring jobs across each of your 100 workspaces seems daunting, the solution is at hand. Through a multi-Workspace deployment, a single job in one Workspace can aggregate data from all specified Workspaces and seamlessly incorporate it into a centralized database in your Lakehouse. This enables you to query the data from any workspace of your choosing, streamlining the monitoring process.</SPAN></P> <H2 class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj"><STRONG class="li fs">Architecture of a multi-Workspace Deployment:</STRONG></H2> <P class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj" data-selectable-paragraph="">Overwatch can be deployed on a single, primary Workspace and then retrieve data from all other Databricks Workspaces. For more details on requirements see<SPAN>&nbsp;</SPAN><A class="af nl" href="https://databrickslabs.github.io/overwatch/deployoverwatch//#multi-workspace-monitoring---considerations" target="_blank" rel="noopener ugc nofollow">Multi-Workspace Consideration</A>. There are many cases where some Workspaces should be able to monitor many Workspaces and others should only monitor themselves. Additionally, co-location of the output data and who should be able to access what data also comes into play, this reference architecture can accommodate all of these needs. To learn more about the details walk through the deployment steps in the <A href="https://databrickslabs.github.io/overwatch/deployoverwatch/" target="_self">official Overwatch documentation</A>.</P> <P class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj" data-selectable-paragraph="">&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="SriramMohanty_11-1701184716367.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/5373i67E4D6D7D1C969B1/image-size/large?v=v2&amp;px=999" role="button" title="SriramMohanty_11-1701184716367.png" alt="SriramMohanty_11-1701184716367.png" /></span></P> <H2 class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj"><STRONG class="li fs">How to perform a multi-Workspace deployment</STRONG></H2> <OL class=""> <LI id="22f9" class="lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me mf mg bj" data-selectable-paragraph="">Download the <A class="af nl" href="https://databrickslabs.github.io/overwatch/assets/DeployOverwatch/072x_overwatch_deployment_config_template.csv" target="_blank" rel="nofollow noopener ugc">CSV</A><SPAN>&nbsp;</SPAN>file.</LI> <LI id="3ef2" class="lg lh fr li b lj ng ll lm ln nh lp lq lr ni lt lu lv nj lx ly lz nk mb mc md me mf mg bj" data-selectable-paragraph="">Fill the CSV with the workspace details which you want to monitor. Please refer the<SPAN>&nbsp;</SPAN><A class="af nl" href="https://databrickslabs.github.io/overwatch/deployoverwatch/configureoverwatch/configuration/#column-description" target="_blank" rel="noopener ugc nofollow">column descriptions</A><SPAN>&nbsp;</SPAN>to know more about the columns.</LI> <LI id="4d82" class="lg lh fr li b lj ng ll lm ln nh lp lq lr ni lt lu lv nj lx ly lz nk mb mc md me mf mg bj" data-selectable-paragraph="">Add<SPAN>&nbsp;</SPAN><A class="af nl" href="https://databrickslabs.github.io/overwatch/deployoverwatch/runningoverwatch/jar/#dependent-library" target="_blank" rel="noopener ugc nofollow">dependant library</A>.</LI> <LI id="16ff" class="lg lh fr li b lj ng ll lm ln nh lp lq lr ni lt lu lv nj lx ly lz nk mb mc md me mf mg bj" data-selectable-paragraph="">Run it via<SPAN>&nbsp;</SPAN><A class="af nl" href="https://databrickslabs.github.io/overwatch/deployoverwatch/runningoverwatch/notebook/#jumpstart-notebook" target="_blank" rel="noopener ugc nofollow">Notebook</A><SPAN>&nbsp;(</SPAN><A class="af nl" href="https://databrickslabs.github.io/overwatch/deployoverwatch/runningoverwatch/notebook/#deployment-example" target="_blank" rel="noopener ugc nofollow">example here</A><SPAN>), or r</SPAN>un it as a<SPAN>&nbsp;</SPAN><A class="af nl" href="https://databrickslabs.github.io/overwatch/deployoverwatch/runningoverwatch/jar/#deploying-overwatch-as-a-jar-on-databricks-workflows" target="_blank" rel="noopener ugc nofollow">JAR</A>.</LI> </OL> <P class="pw-post-body-paragraph lg lh fr li b lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md fk bj" data-selectable-paragraph="">For more details and instructions, please visit the<SPAN>&nbsp;</SPAN><A class="af nl" href="https://databrickslabs.github.io/overwatch/" target="_blank" rel="nofollow noopener ugc">official site</A><SPAN>&nbsp;</SPAN>for Overwatch. You can directly raise an issue in this<SPAN>&nbsp;</SPAN><A class="af nl" href="https://github.com/databrickslabs/overwatch/issues" target="_blank" rel="noopener ugc nofollow">link</A>.</P> Tue, 28 Nov 2023 21:19:01 GMT SriramMohanty 2023-11-28T21:19:01Z https://community.databricks.com/t5/technical-blog/overwatch-the-observability-tool-for-databricks/ba-p/54120 <P><SPAN>Overwatch serves as an Observability tool, offering insights into your cloud expenditures and facilitating cost tracking across diverse dimensions. Whether you are utilising&nbsp;AWS, Azure, or GCP, our robust platform seamlessly integrates with each of these major cloud providers, providing a unified and comprehensive perspective on your cloud costs across various environments. Overwatch stands as a Databricks Labs Project.</SPAN></P> Tue, 28 Nov 2023 21:19:01 GMT https://community.databricks.com/t5/technical-blog/overwatch-the-observability-tool-for-databricks/ba-p/54120 SriramMohanty 2023-11-28T21:19:01Z