cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to get databricks performance metrics programmatically?

Nandhini_Kumar
New Contributor III

How to retrieve all Databricks performance metrics on an hourly basis. Is there a recommended method or API available for retrieving performance metrics ?

4 REPLIES 4

Kaniz_Fatma
Community Manager
Community Manager

Hi @Nandhini_KumarRetrieving performance metrics from Databricks can be done programmatically using various methods.

Here are some approaches:

  1. Databricks UI Metrics:

    • Metrics are available directly in the Databricks UI. You can view them by following these steps:
      1. Click on Compute in the sidebar.
      2. Select the compute resource (cluster) for which you want to view metrics.
      3. By default, hardware metrics are displayed. To view Spark metrics, click the dropdown labelled Hardware and select Spark.
      4. You can also choose GPU if the instance is GPU-enabled1.
  2. Ganglia API:

    • You can query performance data from the Ganglia API on the Spark driver of your Databricks cluster. Hereโ€™s an example in Python:
      import requests
      metric = "mem_report"
      payload = requests.get(f"http://localhost/graph.php?c=cluster&json=1&r=4hr&g={metric}")
      mem_report = payload.json()
      
    • Replace "mem_report" with the specific metric youโ€™re interested in2.
  3. Azure Log Analytics (for Azure Databricks):

Remember to choose the method that best suits your requirements and environment. Happy monitoring! ๐Ÿš€

 

Hi @Kaniz_Fatma 

I encountered the "Error: getaddrinfo ENOTFOUND http" while attempting to run the API.

Could you please provide guidance on resolving this issue? Are there any prerequisites that need to be followed before running the API? I'd appreciate any assistance you can offer.

Nandhini_Kumar
New Contributor III

Thanks @Kaniz_Fatma for your response.
Could you please provide guidance on how we can achieve the step3 in AWS services? 

holly
Valued Contributor III

Hi @Nandhini_Kumar, there's many performance metrics available - it depends on what you're looking to do with this data, and how you look to take action in real time. I would strongly recommend mapping out a user journey so you get only the metrics you need, and don't waste time trying to get these all hourly when they won't be used. 

  • Cluster metrics - you can export these with cluster logging. It's worth noting that ganglia is deprecated for newer runtimes
  • Warehouse metrics - available through the API for query metrics
  • System tables - if you're using unity catalog, this will update hourly
  • Jobs performance - you can use the Jobs API 

API docs can be found here: https://docs.databricks.com/api/workspace/introduction

System tables: https://docs.databricks.com/en/administration-guide/system-tables/index.html

 


 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group