Thanks a lot for the suggestion and for highlighting the cluster-wide listener injection approach using spark.extraListeners. We did consider this option during our initial evaluation and were quite intrigued by its potential to eliminate per-job code changes.
However, we eventually stepped back from it due to the overhead of maintaining a custom JAR especially in a shared environment where changes can have broader implications. For now, we’ve inclined towards the rolling log-based approach as it doesn’t require job-level changes and avoids the overhead of maintaining custom JARs making it more manageable and scalable for our current setup
That said, we’re definitely open to learning more and exploring other strategies and would love to hear more from you or others in the community.