Not able to get spark application in Spark History server using cluster eventlogs

[ossoul]

I'm encountering an issue with incomplete Spark event logs. When I am running the local Spark History Server using the cluster logs, my application appears as "incomplete". Sometime I also see few queries listed as still running, even though the application has completed.

Are there any known issue/limitation with event logging in the Databricks runtime that could cause this? Or is there any specific configuration I should check to ensure complete event logging?

[VZLA]

Thanks for your question!

I believe Databricks has its own SHS implementation, so it's not expected to work with the vanilla SHS. Regarding the queries marked as still running, we can also find this when there are event logs which were not properly closed, or the query failed midway leading to an incomplete/unclosed event.