- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-19-2022 12:12 PM
hi guys,
It`s possible to intercept Spark Listener with Pyspark to collect indicator like shuffle, skew ratio, etc ?
- Labels:
-
Pyspark Spark Listener
-
Sparklistener
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-20-2022 08:30 AM
Hi again! Thank you for your new question! As usual, we'll let your peers have a chance to answer first. Thanks for your patience!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-21-2022 07:04 AM
interesting question.
I know that you can use the SparkListener to collect info, f.e. here.
Mind that the class is written in Scala, so my first thought was that it is not possible in python/pyspark.
But SO says it is possible, but with a lot of overhead tbh.
So far from optimal, I´d try to use scala instead.
