How to send metrics from GCP Databricks to Grafana Cloud through Prometheus?

avinash_goje
New Contributor II

‎06-26-2022 07:04 PM

While connecting the Databricks and Grafana, I have gone through the following approach.

  1. Install Grafna Agent in Databrics Clusters from Databricks console --> Not working since the system is not booted with systemd as init system
  2. Since Spark 3 has Prometheus metrics endpoints, We can use it to add one of the data sources.
    1. Failed due to port opening. (https://dzlab.github.io/bigdata/2020/07/03/spark3-monitoring-1/)

If someone from the community working on a similar problem statement, Please guide. 😊

Labels:

Hubert-Dudek
Databricks MVP

‎06-27-2022 07:32 AM

There is a repo with Prometheus gateway https://gist.github.com/Lowess/3a71792d2d09e38bf8f524644bbf8349.

In the community, we usually use DataDog as both plays nicely https://docs.datadoghq.com/integrations/databricks/?tabs=driveronly


My blog: https://databrickster.medium.com/

jose_gonzalez
Databricks Employee
Databricks Employee
In response to Retired_mod

‎07-29-2022 03:55 PM

Hi @Avinash Goje​,

Just a friendly follow-up. Did you see Hubert's response? did you have time to these these options? please let us know if you still need help or not

0 Kudos