Hi @Brahmareddy ,

I’m doing well, thanks for asking! Hope you’re doing great too.

I’m particularly interested in deep runtime metrics (stage-level, executor-level, and task breakdowns). I actually tried attaching a custom JAR to the cluster for a SparkListener setup, but I couldn’t get it working successfully.

We’re also keen on getting these metrics in near real time rather than only after job completion, since that would help us with monitoring and faster troubleshooting. It would be really helpful if you could guide me through the setup or share a sample configuration that works in Databricks.

Thanks in advance!