how to Intercept Spark Listener with Pyspark ?

[William_Scardua]

hi guys,

It`s possible to intercept Spark Listener with Pyspark to collect indicator like shuffle, skew ratio, etc ?

[Anonymous]

Hi again! Thank you for your new question! As usual, we'll let your peers have a chance to answer first. Thanks for your patience!

[-werners- · Accepted solution]

interesting question.

I know that you can use the SparkListener to collect info, f.e. here.

Mind that the class is written in Scala, so my first thought was that it is not possible in python/pyspark.

But SO says it is possible, but with a lot of overhead tbh.

So far from optimal, I´d try to use scala instead.