How to get databricks performance metrics programmatically?

[Nandhini_Kumar]

How to retrieve all Databricks performance metrics on an hourly basis. Is there a recommended method or API available for retrieving performance metrics ?

[Kaniz_Fatma]

Hi @Nandhini_Kumar, Retrieving performance metrics from Databricks can be done programmatically using various methods.

Here are some approaches:

1. Databricks UI Metrics:

   • Metrics are available directly in the Databricks UI. You can view them by following these steps:
     1. Click on Compute in the sidebar.
     2. Select the compute resource (cluster) for which you want to view metrics.
     3. By default, hardware metrics are displayed. To view Spark metrics, click the dropdown labelled Hardware and select Spark.
     4. You can also choose GPU if the instance is GPU-enabled¹.

2. Ganglia API:

   • You can query performance data from the Ganglia API on the Spark driver of your Databricks cluster. Here’s an example in Python:

     import requests
     metric = "mem_report"
     payload = requests.get(f"http://localhost/graph.php?c=cluster&json=1&r=4hr&g={metric}")
     mem_report = payload.json()
     

   • Replace "mem_report" with the specific metric you’re interested in².

3. Azure Log Analytics (for Azure Databricks):

   • Use Azure Log Analytics to monitor Databricks application job resources. You can send metrics to Azure Log Analytics using methods like Dropwizard or Log4j³.

Remember to choose the method that best suits your requirements and environment. Happy monitoring! 🚀

[Nandhini_Kumar]

Hi @Kaniz_Fatma 

I encountered the "Error: getaddrinfo ENOTFOUND http" while attempting to run the API.

Could you please provide guidance on resolving this issue? Are there any prerequisites that need to be followed before running the API? I'd appreciate any assistance you can offer.

[Nandhini_Kumar]

Thanks @Kaniz_Fatma for your response.
Could you please provide guidance on how we can achieve the step3 in AWS services? 

[holly]

Hi @Nandhini_Kumar, there's many performance metrics available - it depends on what you're looking to do with this data, and how you look to take action in real time. I would strongly recommend mapping out a user journey so you get only the metrics you need, and don't waste time trying to get these all hourly when they won't be used. 

• Cluster metrics - you can export these with cluster logging. It's worth noting that ganglia is deprecated for newer runtimes
• Warehouse metrics - available through the API for query metrics
• System tables - if you're using unity catalog, this will update hourly
• Jobs performance - you can use the Jobs API 

API docs can be found here: https://docs.databricks.com/api/workspace/introduction

System tables: https://docs.databricks.com/en/administration-guide/system-tables/index.html