Hi! 

Today working trying to collect some metrics to create a splot in my spark structure streaming. 

It is configured with a trigger(processingTime="30 seconds") and I am trying to collect data with the following Listener Class (just an example).

 

# Define my listener.

class MyListener(StreamingQueryListener): 

   def onQueryStarted(self, event):

      print("onQueryStarted")

   def onQueryProgress(self, event):

      print("onQueryProgress")

   def onQueryIdle(self, event):

      print("onQueryIdle")

   def onQueryTerminated(self, event):

       print("onQueryTerminated")

# Add my listener.

spark.streams.addListener(MyListener())

 

But it is not working properly, as I show you in the image, it is only catching the event when the streaming starts, but not when each batch is processed.

 

I have tried to write this class in different ways but it didn't work well. Does anybody know what could be the problem here? Does it need an additional configuration?