https://community.databricks.com/t5/data-engineering/collect-spark-ui-statistics-to-analyze-the-performance-for/m-p/73074#M34637 <P>To identify the reasons for a data process poor performance, we need to navigate and analyze the metrics in the Spark UI manually... However, replicating those steps for a giant group of spark applications would be very expensive in times...</P><P>Given this, my question is whether there is a system table in databricks or some strategy that allows me to visualize these metrics for a group of processes and thus be able to realize, in a massive way, how badly my company's processes are performing?</P> Wed, 12 Jun 2024 19:54:57 GMT Mlaricobar94 2024-06-12T19:54:57Z https://community.databricks.com/t5/data-engineering/collect-spark-ui-statistics-to-analyze-the-performance-for/m-p/73074#M34637 <P>To identify the reasons for a data process poor performance, we need to navigate and analyze the metrics in the Spark UI manually... However, replicating those steps for a giant group of spark applications would be very expensive in times...</P><P>Given this, my question is whether there is a system table in databricks or some strategy that allows me to visualize these metrics for a group of processes and thus be able to realize, in a massive way, how badly my company's processes are performing?</P> Wed, 12 Jun 2024 19:54:57 GMT https://community.databricks.com/t5/data-engineering/collect-spark-ui-statistics-to-analyze-the-performance-for/m-p/73074#M34637 Mlaricobar94 2024-06-12T19:54:57Z