topic Re: how to Intercept Spark Listener with Pyspark ? in Data Engineering https://community.databricks.com/t5/data-engineering/how-to-intercept-spark-listener-with-pyspark/m-p/31313#M22795 <P>Hi again! Thank you for your new question! As usual, we'll let your peers have a chance to answer first. Thanks for your patience!</P> Thu, 20 Jan 2022 16:30:20 GMT Anonymous 2022-01-20T16:30:20Z how to Intercept Spark Listener with Pyspark ? https://community.databricks.com/t5/data-engineering/how-to-intercept-spark-listener-with-pyspark/m-p/31312#M22794 <P>hi guys,</P><P>​</P><P>It`s possible to intercept Spark Listener with Pyspark to collect indicator like shuffle, skew ratio, etc ?</P> Wed, 19 Jan 2022 20:12:37 GMT https://community.databricks.com/t5/data-engineering/how-to-intercept-spark-listener-with-pyspark/m-p/31312#M22794 William_Scardua 2022-01-19T20:12:37Z Re: how to Intercept Spark Listener with Pyspark ? https://community.databricks.com/t5/data-engineering/how-to-intercept-spark-listener-with-pyspark/m-p/31313#M22795 <P>Hi again! Thank you for your new question! As usual, we'll let your peers have a chance to answer first. Thanks for your patience!</P> Thu, 20 Jan 2022 16:30:20 GMT https://community.databricks.com/t5/data-engineering/how-to-intercept-spark-listener-with-pyspark/m-p/31313#M22795 Anonymous 2022-01-20T16:30:20Z Re: how to Intercept Spark Listener with Pyspark ? https://community.databricks.com/t5/data-engineering/how-to-intercept-spark-listener-with-pyspark/m-p/31314#M22796 <P>interesting question.</P><P>I know that you can use the SparkListener to collect info, f.e. <A href="https://medium.com/analytics-vidhya/better-spark-logging-and-monitoring-with-spark-listener-c2c5fe86547a" alt="https://medium.com/analytics-vidhya/better-spark-logging-and-monitoring-with-spark-listener-c2c5fe86547a" target="_blank">here</A>.</P><P>Mind that the class is written in Scala, so my first thought was that it is not possible in python/pyspark.</P><P>But <A href="https://stackoverflow.com/questions/44082957/how-to-add-a-sparklistener-from-pyspark-in-python" alt="https://stackoverflow.com/questions/44082957/how-to-add-a-sparklistener-from-pyspark-in-python" target="_blank">SO says it is possible, but with a lot of overhead tbh</A>.</P><P>So far from optimal, I´d try to use scala instead.</P> Fri, 21 Jan 2022 15:04:02 GMT https://community.databricks.com/t5/data-engineering/how-to-intercept-spark-listener-with-pyspark/m-p/31314#M22796 -werners- 2022-01-21T15:04:02Z