How to Get spark history server logs using python REST API

[parimalpatil28]

Hello,

I am trying to load logs from spark job in remote location using Python Rest API.

I want to collect those logs for particular job runs using runID field, that log should contain Errors, Exceptions, and print details.

I have tried "/api/2.1/jobs/runs/get" and "/api/2.1/jobs/runs/get-output" but didnt get desired output logs.

basically I want logs from spark history server.

Let me know what method or call I can use in such case

Thanks in Advance 

Parimal

[Kaniz_Fatma · Accepted solution]

Hi @parimalpatil28, To get the logs from the Spark history server, you can use the Spark REST API. The Spark REST API provides a programmatic interface to access the Spark job, stage, and task information. The API is available at /api/v1/applications/{application_id} endpoint, where {application_id} is the ID of your Spark application.

[parimalpatil28]

Hello,

How can I access endpoint (http URL) of spark history server for a particular cluster?

Thanks,

Parimal  

[Kaniz_Fatma]

Hi @parimalpatil28, To access the endpoint (HTTP URL) of the Spark history server for a particular cluster, you need to identify the cluster-ID first. The cluster ID can be found in the Spark UI URL as mentioned in the provided information.

https://docs.databricks.com/en/clusters/clusters-manage.html