topic Re: Monitoring Azure Databricks in an Azure Log Analytics Workspace in Data Engineering

Monitoring Azure Databricks in an Azure Log Analytics Workspace

Michael_Galli — Fri, 22 Apr 2022 09:46:57 GMT

Does anyone have experience with the mspnp/spark-monitoring library ?

Is this best practice, or are there better ways to monitor a Databricks Cluster?

Re: Monitoring Azure Databricks in an Azure Log Analytics Workspace

Hubert-Dudek — Fri, 22 Apr 2022 10:06:03 GMT

@Michael Galli , Yes, exactly mspnp/spark-monitoring library for Azure Monitor for that https://docs.microsoft.com/en-us/azure/architecture/databricks-monitoring/application-logs

I definitely prefer datadog https://www.datadoghq.com/blog/databricks-monitoring-datadog/. It is simple to integrate and more straightforward to visualize.

Re: Monitoring Azure Databricks in an Azure Log Analytics Workspace

Michael_Galli — Fri, 22 Apr 2022 10:55:36 GMT

@Hubert Dudek Nice... Is it also possible to monitor the metrics collected with mspnp/spark-monitoring in datadog?

Re: Monitoring Azure Databricks in an Azure Log Analytics Workspace

User16764241763 — Mon, 25 Apr 2022 15:38:19 GMT

@Michael Galli I don't think you can monitor metrics captured by mspnp/spark-monitoring in datadog, there is a service called Azure Log Analytics workspace where these logs are available for querying.

You can also check out below if you are interested.

https://databrickslabs.github.io/overwatch/

Re: Monitoring Azure Databricks in an Azure Log Analytics Workspace

Michael_Galli — Tue, 26 Apr 2022 11:25:03 GMT

It´s resolved for me, thanks.

Re: Monitoring Azure Databricks in an Azure Log Analytics Workspace

vr — Thu, 04 Dec 2025 22:50:57 GMT

Interesting that Microsoft deleted this project. Was there any announcement as to when, why, and what to do now?