How to get databricks performance metrics programmatically?

[Nandhini_Kumar]

How to retrieve all Databricks performance metrics on an hourly basis. Is there a recommended method or API available for retrieving performance metrics ?

[Nandhini_Kumar]

Hi @Retired_mod 

I encountered the "Error: getaddrinfo ENOTFOUND http" while attempting to run the API.

Could you please provide guidance on resolving this issue? Are there any prerequisites that need to be followed before running the API? I'd appreciate any assistance you can offer.

[Nandhini_Kumar]

Thanks @Retired_mod for your response.
Could you please provide guidance on how we can achieve the step3 in AWS services? 

[holly]

Hi @Nandhini_Kumar, there's many performance metrics available - it depends on what you're looking to do with this data, and how you look to take action in real time. I would strongly recommend mapping out a user journey so you get only the metrics you need, and don't waste time trying to get these all hourly when they won't be used. 

• Cluster metrics - you can export these with cluster logging. It's worth noting that ganglia is deprecated for newer runtimes
• Warehouse metrics - available through the API for query metrics
• System tables - if you're using unity catalog, this will update hourly
• Jobs performance - you can use the Jobs API 

API docs can be found here: https://docs.databricks.com/api/workspace/introduction

System tables: https://docs.databricks.com/en/administration-guide/system-tables/index.html

[rahuja]

Any response here? Are there any API that expose this? Specially the Job performance metrics?

[holly]

Hi @rahuja  can you be more specific on 'job performance'? Do you mean the spark metrics? Whether it was successful? How much it cost?

[rahuja]

@holly  yes we need spark metrics e.g: amount of Compute used, amount of memory used etc.

[holly]

The spark logs are available through cluster logging. This is enabled at the cluster level for you to choose the destination for the logs. 

Just a heads up - interpreting them at scale is not trivial. I'd recommend having a read through the overwatch project to see how they're aggregated together.