How to Get spark history server logs using python REST API

parimalpatil28 — Tue, 22 Aug 2023 13:17:36 GMT

Hello,

I am trying to load logs from spark job in remote location using Python Rest API.

I want to collect those logs for particular job runs using runID field, that log should contain Errors, Exceptions, and print details.

I have tried "/api/2.1/jobs/runs/get" and "/api/2.1/jobs/runs/get-output" but didnt get desired output logs.

basically I want logs from spark history server.

Let me know what method or call I can use in such case

Thanks in Advance

Parimal

Re: How to Get spark history server logs using python REST API

parimalpatil28 — Thu, 24 Aug 2023 06:14:57 GMT

Hello,

How can I access endpoint (http URL) of spark history server for a particular cluster?

Thanks,

Parimal