Not able to get spark application in Spark History server using cluster eventlogs

ossoul — Tue, 20 Aug 2024 09:44:26 GMT

I'm encountering an issue with incomplete Spark event logs. When I am running the local Spark History Server using the cluster logs, my application appears as "incomplete". Sometime I also see few queries listed as still running, even though the application has completed.

Are there any known issue/limitation with event logging in the Databricks runtime that could cause this? Or is there any specific configuration I should check to ensure complete event logging?

Re: Not able to get spark application in Spark History server using cluster eventlogs

VZLA — Fri, 27 Dec 2024 18:33:50 GMT

Thanks for your question!

I believe Databricks has its own SHS implementation, so it's not expected to work with the vanilla SHS. Regarding the queries marked as still running, we can also find this when there are event logs which were not properly closed, or the query failed midway leading to an incomplete/unclosed event.

topic Not able to get spark application in Spark History server using cluster eventlogs in Data Engineering

Not able to get spark application in Spark History server using cluster eventlogs

Re: Not able to get spark application in Spark History server using cluster eventlogs