How to send metrics from GCP Databricks to Grafana Cloud through Prometheus?

[avinash_goje]

While connecting the Databricks and Grafana, I have gone through the following approach.

1. Install Grafna Agent in Databrics Clusters from Databricks console --> Not working since the system is not booted with systemd as init system
2. Since Spark 3 has Prometheus metrics endpoints, We can use it to add one of the data sources.
   1. Failed due to port opening. (https://dzlab.github.io/bigdata/2020/07/03/spark3-monitoring-1/)

If someone from the community working on a similar problem statement, Please guide. 😊

[Hubert-Dudek]

There is a repo with Prometheus gateway https://gist.github.com/Lowess/3a71792d2d09e38bf8f524644bbf8349.

In the community, we usually use DataDog as both plays nicely https://docs.datadoghq.com/integrations/databricks/?tabs=driveronly

[jose_gonzalez]

Hi @Avinash Goje​,

Just a friendly follow-up. Did you see Hubert's response? did you have time to these these options? please let us know if you still need help or not