REST API

[VMeghraj]

Creating an application to capture cluster metrics and sending HTTP REST request to the Spark History Server's API endpoint to retrieve a list of applications. This request doesn't generate logs in the Spark History Server's log files. The Spark History Server logs primarily capture information related to the Spark applications it serves and its own internal operation, not external API requests.

Ref : https://spark.apache.org/docs/latest/monitoring.html#rest-api

1) Sample REST request using command.
curl http://<hostname>:18080/api/v1/applications/

2) We should get the expected JSON response.

3) Check the History server logs /opt/mapr/spark/spark-2.4.8/logs/spark-mapr-org.apache.spark.deploy.history.HistoryServer*.out and we do NOT see any logging statements for the request #1.

Where does the REST api requests get's logged ?

[Kaniz_Fatma]

Hi @VMeghraj , The REST API requests do not get logged in the Spark History Server's log files. However, they can be logged on the client side, i.e., the application that is sending the HTTP REST requests. This can be done by implementing logging in the application code itself.

[VMeghraj]

Do you have any examples how it can be logged on client side ? Enabling verbose logging on curl request dosen't help much.

e.g) curl -v http://<hostname>:18080/api/v1/applications/