Is it possible to obtain a job's event log via the...

mrstevegross · ‎01-15-2025

Currently, to investigate job performance, I can look at a job's information (via the UI) to see the "Event Log" (pictured below):

I'd like to obtain this information programmatically, so I can analyze it across jobs. However, the docs for the `get` call (https://docs.databricks.com/api/workspace/jobs/get) do not appear to include information about the event log. Is there a way to get this info via the REST API?

Alberto_Umana · ‎01-15-2025

Hi @mrstevegross,

Unfortunately the API job/get does not have event logs in its output. I will see if there is a workaround, but as far as I can tell from REST API might no be possible.

View solution in original post

mrstevegross · ‎01-15-2025

Darn, well, good to know. Can y'all consider this a feature request in that case?

Alberto_Umana · ‎01-15-2025

Can you give me more context on why do you need event logs?

mrstevegross · ‎01-15-2025

Sure, I want to assess the overall performance of our Spark jobs, particularly the time between "CREATING" and "RUNNING". It's very time-consuming to gather this data manually via the UI; if there is a way to get it programmatically that would be great.

mrstevegross · ‎02-25-2025

Followup: I see that there is documentation on exporting Spark logs (https://docs.databricks.com/aws/en/compute/clusters-manage#event-log) and init script logging (https://docs.databricks.com/aws/en/init-scripts/logs#init-script-events). I've been able to export the spark logs itself, but I'm not seeing the job event information there. Is there some way to gather the event log via these mechanisms?

mrstevegross · ‎02-25-2025

I also see there is a "list cluster events" API (https://docs.databricks.com/api/workspace/clusters/events); can I get the event log this way?

Is it possible to obtain a job's event log via the REST API?