Monitoring Databricks

[SainnathReddy_M]

Good Day All,

 Did any one done Databricks Monitoring integration with  any other 3rd party applications like Grafana or ELK to get infrastructure monitoring like cpu_utilization ,memory ,job monitor and I can able to write spark code to get cpu_utilization and I am able to view that in Grafana but not in ELK,Instead of this can we push entire data from databricks to ELK or Grafana 

[szymon_dybczak]

I didn't test it myself, but for sure there is such a capability. There is Grafana plugin for databricks (link to documentation below):
Databricks datasource for Grafana | Grafana Enterprise Plugins documentation

Here, example how to use it
Monitoring Your DataBricks Clusters with Grafana: How to Push Metrics and Logs for Maximum Efficienc...

If you are using Azure you can also consider sending data to Azure Monitor

Monitor Azure Databricks - Azure Architecture Center | Microsoft Learn

[Rishabh_Tiwari]

Hi @SainnathReddy_M ,

Thank you for reaching out to our community! We're here to help you. 

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.

If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!

Thanks,

Rishabh