Dashboard

subhas_1729 — Mon, 17 Feb 2025 00:46:29 GMT

I want to design a dashboard that will show some variables of Spark-UI. Is it possible to access Spark-UI variables from my spark program.

Re: Dashboard

Alberto_Umana — Mon, 17 Feb 2025 02:11:29 GMT

Hi @subhas_1729,

You can achieve this by leveraging Spark's monitoring and instrumentation APIs.

Spark provides metrics that can be accessed through the SparkListener interface as well as the REST API. The SparkListener interface allows you to receive notifications about events happening in your Spark application, such as job start/end and stage start/end. By implementing SparkListener and registering it with your SparkContext, you can collect metrics and other information that are typically displayed in the Spark UI

Implementing these approaches will allow your Spark program to access the required metrics and variables to design an informative dashboard similar to the Spark UI.

https://spark.apache.org/docs/latest/monitoring.html

topic Dashboard in Data Engineering

Dashboard

Re: Dashboard