Hello @saicharandeepb 

I would recommend to use Gist by rayalex 

It integrates EC2 Alloy with Prometheus and Grafana, allowing you to capture and visualize Spark run-time and structured streaming metrics in near real-time.

It’s not a solution natively integrated in Databricks (since, as far as I know, runtime-level access is restricted), but I think it’s a very solid approach if your goal is to collect this information and display it in a dashboard.

Hope this helps 🙂

Isi