https://community.databricks.com/t5/administration-architecture/monitor-and-alert-databricks-resource-utilization-and-cost/m-p/61155#M895 <P>We want to build monitoring and Alerting solution for Azure Databricks that should capture Resource Utilization details (like Aggregated CPU%, Memory% etc.) and Cost consumption at the Account Level.</P><P>We have Unity Catalog Enabled and there are multiple workspaces associated.</P><P>We prefer utilizing Diagnostics on Azure Log Analytics Workspace, as we do not want additional cost overhead. Diagnostic setting is enabled for Clusters, Jobs, UC, Notebooks, Accounts etc.</P><P>What are various KPIs and Metrics to consider? Please share the evaluation of the KPIs</P><P>What are the ways Alerting can be implemented using Log Analytics for Databricks</P> Mon, 19 Feb 2024 17:07:39 GMT smehta_0908 2024-02-19T17:07:39Z https://community.databricks.com/t5/administration-architecture/monitor-and-alert-databricks-resource-utilization-and-cost/m-p/61155#M895 <P>We want to build monitoring and Alerting solution for Azure Databricks that should capture Resource Utilization details (like Aggregated CPU%, Memory% etc.) and Cost consumption at the Account Level.</P><P>We have Unity Catalog Enabled and there are multiple workspaces associated.</P><P>We prefer utilizing Diagnostics on Azure Log Analytics Workspace, as we do not want additional cost overhead. Diagnostic setting is enabled for Clusters, Jobs, UC, Notebooks, Accounts etc.</P><P>What are various KPIs and Metrics to consider? Please share the evaluation of the KPIs</P><P>What are the ways Alerting can be implemented using Log Analytics for Databricks</P> Mon, 19 Feb 2024 17:07:39 GMT https://community.databricks.com/t5/administration-architecture/monitor-and-alert-databricks-resource-utilization-and-cost/m-p/61155#M895 smehta_0908 2024-02-19T17:07:39Z https://community.databricks.com/t5/administration-architecture/monitor-and-alert-databricks-resource-utilization-and-cost/m-p/61202#M897 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/91252">@smehta_0908</a>&nbsp;Greetings!</P> <P>You can utilize Datadog for monitoring CPU and memory of clusters.</P> <P><A href="https://docs.datadoghq.com/integrations/databricks/?tab=driveronly" target="_blank">https://docs.datadoghq.com/integrations/databricks/?tab=driveronly</A></P> <P>For Cost consumption at accounts level you can make use of&nbsp;<SPAN>billable usage logs using the&nbsp;</SPAN><A class="reference external" href="https://docs.databricks.com/api/account/billableusage/download" target="_blank" rel="noopener">Account API</A></P> <P><A href="https://docs.databricks.com/en/administration-guide/account-settings/usage.html" target="_blank">https://docs.databricks.com/en/administration-guide/account-settings/usage.html</A></P> Tue, 20 Feb 2024 08:05:22 GMT https://community.databricks.com/t5/administration-architecture/monitor-and-alert-databricks-resource-utilization-and-cost/m-p/61202#M897 Allia 2024-02-20T08:05:22Z