Monitoring Azure Databricks in an Azure Log Analytics Workspace

[Michael_Galli]

Does anyone have experience with the mspnp/spark-monitoring library ?

Is this best practice, or are there better ways to monitor a Databricks Cluster?

[Hubert-Dudek · Accepted solution]

@Michael Galli​ , Yes, exactly mspnp/spark-monitoring library for Azure Monitor for that https://docs.microsoft.com/en-us/azure/architecture/databricks-monitoring/application-logs

I definitely prefer datadog https://www.datadoghq.com/blog/databricks-monitoring-datadog/. It is simple to integrate and more straightforward to visualize.

[Michael_Galli]

@Hubert Dudek​ Nice... Is it also possible to monitor the metrics collected with mspnp/spark-monitoring in datadog?

[Kaniz]

Hi @Michael Galli​ , For more information about using this library to monitor Azure Databricks, see Monitoring Azure Databricks.

Please let us know if this helps, or we'll find another answer for you.

[User16764241763]

@Michael Galli​  I don't think you can monitor metrics captured by mspnp/spark-monitoring in datadog, there is a service called Azure Log Analytics workspace where these logs are available for querying.

You can also check out below if you are interested.

https://databrickslabs.github.io/overwatch/

[Kaniz]

Hi @Michael Galli​ , Would you be interested in discussing more on it, or did @Arvind Ravish​ 's response help resolve your query?

[Michael_Galli]

It´s resolved for me, thanks.

[Kaniz]

Hi @Michael Galli​ , Thanks for the quick update. Would you mind marking an answer as the best to help the community?