We want to build monitoring and Alerting solution for Azure Databricks that should capture Resource Utilization details (like Aggregated CPU%, Memory% etc.) and Cost consumption at the Account Level.
We have Unity Catalog Enabled and there are multiple workspaces associated.
We prefer utilizing Diagnostics on Azure Log Analytics Workspace, as we do not want additional cost overhead. Diagnostic setting is enabled for Clusters, Jobs, UC, Notebooks, Accounts etc.
What are various KPIs and Metrics to consider? Please share the evaluation of the KPIs
What are the ways Alerting can be implemented using Log Analytics for Databricks
