How to send metrics from GCP Databricks to Grafana Cloud through Prometheus?

[avinash_goje]

While connecting the Databricks and Grafana, I have gone through the following approach.

1. Install Grafna Agent in Databrics Clusters from Databricks console --> Not working since the system is not booted with systemd as init system
2. Since Spark 3 has Prometheus metrics endpoints, We can use it to add one of the data sources.
   1. Failed due to port opening. (https://dzlab.github.io/bigdata/2020/07/03/spark3-monitoring-1/)

If someone from the community working on a similar problem statement, Please guide. 😊

[Hubert-Dudek]

There is a repo with Prometheus gateway https://gist.github.com/Lowess/3a71792d2d09e38bf8f524644bbf8349.

In the community, we usually use DataDog as both plays nicely https://docs.datadoghq.com/integrations/databricks/?tabs=driveronly

[Kaniz_Fatma]

Hi @Avinash Goje​, We haven’t heard from you on the last response from @Hubert Dudek​ , and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the community as it can be helpful to others.

[jose_gonzalez]

Hi @Avinash Goje​,

Just a friendly follow-up. Did you see Hubert's response? did you have time to these these options? please let us know if you still need help or not