topic Re: How to get databricks performance metrics programmatically? in Get Started Discussions

How to get databricks performance metrics programmatically?

Nandhini_Kumar — Tue, 02 Apr 2024 04:52:38 GMT

How to retrieve all Databricks performance metrics on an hourly basis. Is there a recommended method or API available for retrieving performance metrics ?

Re: How to get databricks performance metrics programmatically?

Nandhini_Kumar — Tue, 02 Apr 2024 12:38:39 GMT

Thanks @Retired_mod for your response.
Could you please provide guidance on how we can achieve the step3 in AWS services?

Re: How to get databricks performance metrics programmatically?

Nandhini_Kumar — Wed, 03 Apr 2024 02:31:33 GMT

Hi @Retired_mod

I encountered the "Error: getaddrinfo ENOTFOUND http" while attempting to run the API.

Could you please provide guidance on resolving this issue? Are there any prerequisites that need to be followed before running the API? I'd appreciate any assistance you can offer.

Re: How to get databricks performance metrics programmatically?

holly — Wed, 03 Apr 2024 09:02:53 GMT

Hi @Nandhini_Kumar, there's many performance metrics available - it depends on what you're looking to do with this data, and how you look to take action in real time. I would strongly recommend mapping out a user journey so you get only the metrics you need, and don't waste time trying to get these all hourly when they won't be used.

Cluster metrics - you can export these with cluster logging. It's worth noting that ganglia is deprecated for newer runtimes
Warehouse metrics - available through the API for query metrics
System tables - if you're using unity catalog, this will update hourly
Jobs performance - you can use the Jobs API

API docs can be found here: https://docs.databricks.com/api/workspace/introduction

System tables: https://docs.databricks.com/en/administration-guide/system-tables/index.html

Re: How to get databricks performance metrics programmatically?

rahuja — Wed, 06 Nov 2024 13:51:19 GMT

Any response here? Are there any API that expose this? Specially the Job performance metrics?

Re: How to get databricks performance metrics programmatically?

holly — Mon, 18 Nov 2024 10:59:22 GMT

Hi @rahuja can you be more specific on 'job performance'? Do you mean the spark metrics? Whether it was successful? How much it cost?

Re: How to get databricks performance metrics programmatically?

rahuja — Mon, 18 Nov 2024 12:34:28 GMT

@holly yes we need spark metrics e.g: amount of Compute used, amount of memory used etc.

Re: How to get databricks performance metrics programmatically?

holly — Thu, 21 Nov 2024 14:19:46 GMT

The spark logs are available through cluster logging. This is enabled at the cluster level for you to choose the destination for the logs.

Just a heads up - interpreting them at scale is not trivial. I'd recommend having a read through the overwatch project to see how they're aggregated together.