Hey,
So currently I'm using Pyspark and utilizing SparkListener to track metrics of my spark jobs. The problem I can't seem to solve is why in databricks the listener cannot be removed with removeSparkListener as it still is attached to the context.
# Func--
def unregister(spark_context: SparkContext, *listeners: SparkListener):
for listener in listeners:
spark_context._jsc.sc().removeSparkListener(listener)
#spark_context._gateway.shutdown_callback_server()
# Single--
sc._jsc.sc().removeSparkListener(listener) # Where SC is the SparkContext
