cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Monitor and Alert Databricks Resource Utilization and Cost Consumption

smehta_0908
New Contributor II

We want to build monitoring and Alerting solution for Azure Databricks that should capture Resource Utilization details (like Aggregated CPU%, Memory% etc.) and Cost consumption at the Account Level.

We have Unity Catalog Enabled and there are multiple workspaces associated.

We prefer utilizing Diagnostics on Azure Log Analytics Workspace, as we do not want additional cost overhead. Diagnostic setting is enabled for Clusters, Jobs, UC, Notebooks, Accounts etc.

What are various KPIs and Metrics to consider? Please share the evaluation of the KPIs

What are the ways Alerting can be implemented using Log Analytics for Databricks

1 REPLY 1

AlliaKhosla
New Contributor III
New Contributor III

@smehta_0908 Greetings!

You can utilize Datadog for monitoring CPU and memory of clusters.

https://docs.datadoghq.com/integrations/databricks/?tab=driveronly

For Cost consumption at accounts level you can make use of billable usage logs using the Account API

https://docs.databricks.com/en/administration-guide/account-settings/usage.html

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!