Monitoring Azure Databricks in an Azure Log Analytics Workspace

[Michael_Galli]

Does anyone have experience with the mspnp/spark-monitoring library ?

Is this best practice, or are there better ways to monitor a Databricks Cluster?

[Hubert-Dudek · Accepted solution]

@Michael Galli​ , Yes, exactly mspnp/spark-monitoring library for Azure Monitor for that https://docs.microsoft.com/en-us/azure/architecture/databricks-monitoring/application-logs

I definitely prefer datadog https://www.datadoghq.com/blog/databricks-monitoring-datadog/. It is simple to integrate and more straightforward to visualize.

[Michael_Galli]

@Hubert Dudek​ Nice... Is it also possible to monitor the metrics collected with mspnp/spark-monitoring in datadog?

[User16764241763]

@Michael Galli​  I don't think you can monitor metrics captured by mspnp/spark-monitoring in datadog, there is a service called Azure Log Analytics workspace where these logs are available for querying.

You can also check out below if you are interested.

https://databrickslabs.github.io/overwatch/

[Michael_Galli]

It´s resolved for me, thanks.

[vr]

Interesting that Microsoft deleted this project. Was there any announcement as to when, why, and what to do now?