Hi @saicharandeepb 

Good job on doing such detailed research on monitoring structured streaming. If you need lower latency than rolling log permits, then have you tried this:
Cluster-wide listener injection: Use spark.extraListeners to register a custom listener JAR that emits StreamingQueryProgress metrics directly to your sink (Event Hubs, Delta). This avoids per-job code changes but does alter cluster config and requires packaging and maintenance. This will inject a cluster-wide listener using spark.extraListeners so every job on the cluster automatically emits Structured Streaming StreamingQueryProgress metrics to your sink (Event Hubs, Delta, etc.).
Pros: no per-streaming code changes.
Cons: you’re changing cluster behavior for all jobs and you now own a JAR + config.