Unregistering / Removing A SparkListener

[Scorpius]

Hey,

So currently I'm using Pyspark and utilizing SparkListener to track metrics of my spark jobs. The problem I can't seem to solve is why in databricks the listener cannot be removed with removeSparkListener as it still is attached to the context.

# Func--
def unregister(spark_context: SparkContext, *listeners: SparkListener):
    for listener in listeners:
      spark_context._jsc.sc().removeSparkListener(listener)
    #spark_context._gateway.shutdown_callback_server()
 
 
 
 
 
# Single--
sc._jsc.sc().removeSparkListener(listener) # Where SC is the SparkContext

[Scorpius]

Thanks for the reply!

Although in practice this should work, but it doesn't work if it's running in Databricks. Is there any other readings you can provide?