I'm encountering an issue with incomplete Spark event logs. When I am running the local Spark History Server using the cluster logs, my application appears as "incomplete". Sometime I also see few queries listed as still running, even though the application has completed.
Are there any known issue/limitation with event logging in the Databricks runtime that could cause this? Or is there any specific configuration I should check to ensure complete event logging?
