https://community.databricks.com/t5/data-engineering/unregistering-removing-a-sparklistener/m-p/16996#M11077 <P>Thanks for the reply!</P><P></P><P>Although in practice this should work, but it doesn't work if it's running in Databricks. Is there any other readings you can provide?</P> Fri, 24 Jun 2022 19:56:07 GMT Scorpius 2022-06-24T19:56:07Z https://community.databricks.com/t5/data-engineering/unregistering-removing-a-sparklistener/m-p/16994#M11075 <P>Hey, </P><P></P><P>So currently I'm using Pyspark and utilizing SparkListener&nbsp;to track metrics of my spark jobs. The problem I can't seem to solve is why in databricks the listener cannot be removed with removeSparkListener as it still is attached to the context.</P><P></P><PRE><CODE># Func-- def unregister(spark_context: SparkContext, *listeners: SparkListener): for listener in listeners: spark_context._jsc.sc().removeSparkListener(listener) #spark_context._gateway.shutdown_callback_server() &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; # Single-- sc._jsc.sc().removeSparkListener(listener) # Where SC is the SparkContext</CODE></PRE><P></P> Tue, 21 Jun 2022 22:26:34 GMT https://community.databricks.com/t5/data-engineering/unregistering-removing-a-sparklistener/m-p/16994#M11075 Scorpius 2022-06-21T22:26:34Z https://community.databricks.com/t5/data-engineering/unregistering-removing-a-sparklistener/m-p/16996#M11077 <P>Thanks for the reply!</P><P></P><P>Although in practice this should work, but it doesn't work if it's running in Databricks. Is there any other readings you can provide?</P> Fri, 24 Jun 2022 19:56:07 GMT https://community.databricks.com/t5/data-engineering/unregistering-removing-a-sparklistener/m-p/16996#M11077 Scorpius 2022-06-24T19:56:07Z