topic REST API for Stream Monitoring in Data Engineering

REST API for Stream Monitoring

Baldrez — Thu, 14 Oct 2021 13:42:04 GMT

Hi, everyone. I just recently started using Databricks on Azure so my question is probably very basic but I am really stuck right now.

I need to capture some streaming metrics (number of input rows and their time) so I tried using the Spark Rest Api , however I get the following error: "no streaming listener attached to Databricks Shell". I tried different solutions I have seen in videos or tutorials but none have worked so far (this only happens when I try to get the stream statistics, if I use the API for jobs or stages, I get the json as expected).

Here is the code I am trying to run:

import requests
import json
 
driverIp = spark.conf.get('spark.driver.host')
port = spark.conf.get("spark.ui.port")
temp_url = F"http://{driverIp}:{port}/api/v1/applications"
temp_r = request.get(temp_url, timeout=10.0)
content_r = json.load(temp_r.content)
app_id = content_r[0][ïd"]
 
url = F"http://{driverIp}:{port}/api/v1/applications/{app_id}/streaming/statistics"
r = requests.get(url)
print(r.content)

I understand that I should attach the streaming listener in order to get the metrics I need but I still did not understand how to implement it in the code. Could someone please help me on this issue?

Thanks a lot in advance

Re: REST API for Stream Monitoring

Anonymous — Fri, 15 Oct 2021 16:28:23 GMT

Hi @Roberto Baldrez - My name is Piper and I'm one of the community moderators. Thanks for your question. Let's give it a bit to see what the community says. Thank you for your patience.

Re: REST API for Stream Monitoring

User16763506477 — Wed, 20 Oct 2021 23:23:46 GMT

Hi @Roberto Baldrez , you will need to add the below configs to the cluster

spark.sql.streaming.metricsEnabled true
*.sink.servlet.class org.apache.spark.metrics.sink.MetricsServlet
*.sink.servlet.path /metrics/json
master.sink.servlet.path /metrics/master/json
applications.sink.servlet.path /metrics/applications/json

URL will change to "http://<driverIP>:<port>/metrics/json/" the one you mentioned is for DStream application

note: This gives limited streaming metrics. If you need all metrics you will need to add metrics sink to the cluster.

More info

Re: REST API for Stream Monitoring

jose_gonzalez — Tue, 26 Oct 2021 23:55:31 GMT

hi @Roberto Baldrez ,

if you think that @Gaurav Rupnar solved your question, then please select it as best response to it can be moved to the top of the topic and it will help more users in the future.

Thank you

Re: REST API for Stream Monitoring

lixing — Wed, 21 Feb 2024 17:08:22 GMT

Could you please tell us where is the configs to the cluster? I cannot find it. Thanks.