Is it possible to view Databricks cluster metrics using REST API

[Prashanthkumar]

I am looking for some help on getting databricks cluster metrics such as memory utilization, CPU utilization, memory swap utilization, free file system using REST API.

I am trying it in postman using databricks token and with my Service Principal bearer token with this link

https://databricksinstance.azuredatabricks.net/?o=workspaceid#setting/clusters/cluster_id/metrics

however when i send the request it shows below sign in, is there anything else i am missing or is it not possible to view using REST API

Here is the full response.

Any suggestions?

[Walter_C]

There is currently no option available to get this metrics available through API, but is coming soon.

[Prashanthkumar]

Thank you Walter will keep an eye for updates.

[ErikJ]

Is there a release calender where I can find a date for this?

[Nandhini_Kumar]

Is there any alternative way to get all performance metrics programmatically?

[ErikJ]

Through Databricks REST API you can get job info such as pipeline run time, task run times etc, but I don't think you can get CPU usage and such

[Prashanthkumar]

OK thank you, any plans to introduce a new feature in Databricks to capture CPU usage?

[javierbg]

At my company we are also interested in this feature, is there an ETA?

[moto-charles]

My company is also interested in this feature and interested in a roadmap or ETA.

[Walter_C]

There is no ETA yet available, as a temporary workaround you can get metrics details via the system tables through the node_timeline

https://docs.databricks.com/en/admin/system-tables/compute.html#node-timeline-table-schema 

[makulr]

Hard to believe this isnt a feature yet. I mean if I can go to the cluster, click metrics and see them, why can't I get them via API?? 

I guess I'll go with the workaround, but this is embarrassing honestly.

[Walter_C]

We understand and our product team is working on this, our direction has been to go through the system tables for this information but we are exploring in giving additional options

[Prashanthkumar]

Thank you Walter, will keep an eye.

[aranjan99]

Is there any timeline on when will this get delivered? System tables does not give data for all clusters. Any clusters terminated within 10 minutes do not show up in that table. Its quite confusing to have the data for some clusters and missing for some

[Walter_C]

As of now there is no ETA yet available