Hey. I'm working on a project where I'd like to be able to view and play around with the spark cluster metrics. I'd like to know what the utilization % and max values are for metrics like CPU, memory and network. I've tried using some open source solutions(https://github.com/mspnp/spark-monitoring) but I'm not really getting what I'm looking for. Ideally, a solution that could give me insights on my Azure Databricks instances to optimize usage would be perfect. Currently, I can access some of these metrics on the metrics tab on my spark cluster page as static images but it'd be great if I could export that information to make my own insights or graphs.
