Fetching CPU and memory data using REST APIs

[Neelofer]

Hi,

I am trying to fetch CPU and memory details from Databricks. Are there any APIs present to which I can connect using postman and fetch these details?

[Kaniz_Fatma]

Hi @Neelofer, To fetch CPU and memory details from Databricks, you can utilize the Databricks REST APIs.

1. While it doesn’t provide real-time data, the Spark UI offers information related to Spark-specific performance, such as job and task breakdowns. However, it’s not specifically designed for CPU and memory monitoring¹.

2. Ganglia metrics can give you real-time and historical data on CPU, disk, and memory usage. Keep in mind that these metrics are more general and not specific to Databricks alone.

3. Databricks Cluster Metrics: Available in all Databricks runtimes, cluster metrics allow you to inspe...².

4. Datadog Integration: You can also use Datadog for monitoring CPU and memory of Databricks clusters³.

Remember to check the Databricks REST API reference for detailed information on available endpoints ...⁴.

Happy querying! 😊