Databricks Observability

[Ashay]

Hi

I learned about databricks observability platform during Data+AI summit 2024. There was also a session around it https://www.databricks.com/dataaisummit/session/simplifying-lakehouse-observability-databricks-key-d.... However, I am not able to find any documentation around it or list of comprehensive metrics which this platform is supposed to capture. Can someone point me in the right direction to gather this information?

Thanks

Ashay

[Kaniz_Fatma]

Hi @Ashay, It’s great that you attended the Data+AI Summit 2024! The Databricks observability platform, often referred to as Overwatch, is designed to help monitor and optimize your Databricks environment. Here are some resources and key points to get you started:

1. Overwatch Documentation: Overwatch is an observability tool that collects job and audit log data, integrates it with data from the Databricks REST API, and processes it into tables that describe the ongoing activity of your Databricks workspaces. You can find more details about Overwatch and its features in the Databricks Community Technical Blog.

2. Databricks Lakehouse Monitoring: This tool allows you to monitor the statistical properties and quality of data in your tables, track the performance of machine learning models, and monitor inference tables. You can learn more about its components and usage in the Databricks Lakehouse Monitoring documentation.

3. Comprehensive Guide: For a deeper dive into data observability within the Databricks platform, inclu...Orchestra guide.

These resources should provide you with a comprehensive understanding of the metrics and capabilities of the Databricks observability platform.

If you have any specific questions or need further assistance, feel free to ask!