https://community.databricks.com/t5/data-engineering/forward-spark-structured-streaming-metrics-to-datadog/m-p/30782#M22349 <P>We have a spark streaming application written in Pyspark that we'd like to monitor with Datadog. By default, datadog collects a couple of streaming metrics like '<I>spark.structured_streaming.processing_rate</I>' and '<I>spark.structured_streaming.latency</I>'. However, after setting '<I>logs_enabled: true</I>' and '<I>spark.sql.streaming.metricsEnabled = true</I>' in the cluster init script. We're still unable to see any streaming metrics in datadog. Upon some research, it seems like we need to implement a new class of '<I>StreamingQueryListener</I>' from spark streaming to make this work. Is this assumption correct? If so, is it possible to implement this in Python instead of Scala? I haven't seen any Python implementation anywhere. I would appreciate it if someone can point me to any example if it's possible. Any help would be appreciated!</P> Mon, 26 Sep 2022 22:34:44 GMT Lizzz 2022-09-26T22:34:44Z https://community.databricks.com/t5/data-engineering/forward-spark-structured-streaming-metrics-to-datadog/m-p/30782#M22349 <P>We have a spark streaming application written in Pyspark that we'd like to monitor with Datadog. By default, datadog collects a couple of streaming metrics like '<I>spark.structured_streaming.processing_rate</I>' and '<I>spark.structured_streaming.latency</I>'. However, after setting '<I>logs_enabled: true</I>' and '<I>spark.sql.streaming.metricsEnabled = true</I>' in the cluster init script. We're still unable to see any streaming metrics in datadog. Upon some research, it seems like we need to implement a new class of '<I>StreamingQueryListener</I>' from spark streaming to make this work. Is this assumption correct? If so, is it possible to implement this in Python instead of Scala? I haven't seen any Python implementation anywhere. I would appreciate it if someone can point me to any example if it's possible. Any help would be appreciated!</P> Mon, 26 Sep 2022 22:34:44 GMT https://community.databricks.com/t5/data-engineering/forward-spark-structured-streaming-metrics-to-datadog/m-p/30782#M22349 Lizzz 2022-09-26T22:34:44Z https://community.databricks.com/t5/data-engineering/forward-spark-structured-streaming-metrics-to-datadog/m-p/30783#M22350 <P>@Liz Zhang​&nbsp;, Please refer to the below documentation contain pyspark implementation of streamingQueryListener </P><P><A href="https://www.databricks.com/blog/2022/05/27/how-to-monitor-streaming-queries-in-pyspark.html" target="test_blank">https://www.databricks.com/blog/2022/05/27/how-to-monitor-streaming-queries-in-pyspark.html</A></P> Tue, 27 Sep 2022 15:48:52 GMT https://community.databricks.com/t5/data-engineering/forward-spark-structured-streaming-metrics-to-datadog/m-p/30783#M22350 shan_chandra 2022-09-27T15:48:52Z