Hi all,

I’m trying to get Spark run-time metrics and structured streaming metrics by enabling cluster logging and now I see the following folders:

What I noticed is that the eventlog folder only gets populated after a job has completed. That makes it difficult to calculate metrics in near real-time.

Is there a common parser or recommended approach to read from the driver and executor logs so that I can compute these metrics while the job is still running, rather than only after completion?

Thanks in advance for your guidance!