cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Dashboard

subhas_1729
New Contributor II

Hi

       I want to design a dashboard that will show some variables of Spark-UI. Is it possible to access Spark-UI variables from my spark program.

 

1 REPLY 1

Alberto_Umana
Databricks Employee
Databricks Employee

Hi @subhas_1729,

You can achieve this by leveraging Spark's monitoring and instrumentation APIs.

Spark provides metrics that can be accessed through the SparkListener interface as well as the REST API. The SparkListener interface allows you to receive notifications about events happening in your Spark application, such as job start/end and stage start/end. By implementing SparkListener and registering it with your SparkContext, you can collect metrics and other information that are typically displayed in the Spark UI

Implementing these approaches will allow your Spark program to access the required metrics and variables to design an informative dashboard similar to the Spark UI.

https://spark.apache.org/docs/latest/monitoring.html

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now