spark cluster monitoring and visibility
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-16-2022 12:33 AM
Hey. I'm working on a project where I'd like to be able to view and play around with the spark cluster metrics. I'd like to know what the utilization % and max values are for metrics like CPU, memory and network. I've tried using some open source solutions(https://github.com/mspnp/spark-monitoring) but I'm not really getting what I'm looking for. Ideally, a solution that could give me insights on my Azure Databricks instances to optimize usage would be perfect. Currently, I can access some of these metrics on the metrics tab on my spark cluster page as static images but it'd be great if I could export that information to make my own insights or graphs.
Labels:
- Labels:
-
Azure
-
Spark Cluster
-
Spark monitoring