topic Re: Grafana + databricks = True? in Data Engineering

Grafana + databricks = True?

Erik — Thu, 16 Dec 2021 19:23:08 GMT

We have some timeseries in databricks, and we are reading them into powerbi through sql compute endpoints. For timeseries powerbi is ... not optimal. Earlier I have used grafana with various backends, and quite like it, but I cant find any way to connect grafana with databricks. So I have two questions:

Does anyone know if it is possible to use databricks/spark as a datasource for grafana in some way (I have ofcourse done the normal googlesearch)?
Is there any other good way to query, visualize, and explore spark/databricks timeseries?

Re: Grafana + databricks = True?

wrnjun — Tue, 21 Dec 2021 11:31:13 GMT

I was going to migrate all my data to Databricks, but Databricks does not support Grafana

Re: Grafana + databricks = True?

André — Thu, 03 Mar 2022 21:40:40 GMT

Hi,

@Kaniz Fatma is there any update on this? I understand this is available: https://databrickslabs.github.io/overwatch/

Although it doesn't have official support which might be an issue. Thank you.

Re: Grafana + databricks = True?

Erik — Mon, 07 Mar 2022 11:18:18 GMT

@André Monteiro Unfortunately I think Overwatch uses influx somewhere in there as the backend for those dashboards (if I remember correctly from the presentation @Alex Ott had for our company, maybe you can confirm?) But it would definitely be cooler if spark could be the backend;-)

Re: Grafana + databricks = True?

alexott — Mon, 07 Mar 2022 11:46:50 GMT

Overwatch uses Delta to store all the data, so you can use Databricks notebooks or Databricks SQL for generation of dashboards. Influx was mentioned in context of more real-time implementation

Re: Grafana + databricks = True?

Erik — Mon, 07 Mar 2022 12:32:21 GMT

@Alex Ott : But the pretty graphs at the bottom of https://databrickslabs.github.io/overwatch/, which looks very much like grafana, they are not backed by spark right (i.e. you guys have not made a spark grafana-datasource as part of overwatch)?

Imagine how cool it would be with a grafana dbr-sql datasource, and you could get real-time info directly from your delta-table into grafana! If we only knew someone proficent in both Go and dbr-sql 😉

Re: Grafana + databricks = True?

alexott — Mon, 07 Mar 2022 13:02:59 GMT

Yes, the screenshots at bottom are for the realtime version, but it requires quite a lot of effort. Regarding Grafana - I recommend to talk with your CSE or SA - maybe it could be prioritized

Re: Grafana + databricks = True?

Erik — Tue, 05 Jul 2022 08:34:44 GMT

So, just to update people, since things have changed a bit! The good news is that Grafana has released a databricks data-connector! The bad news is that its "enterprise only". I have played a bit with it, and it works quite well. Here is an example snippet which fetches a timeseries given the time-bounds given in grafana, and resamples it to the bucket-size grafana proposes:

SELECT window.start as time, avg(value),
FROM
mydb.mytable
WHERE timestamp >= "${__from:date:iso}" and timestamp <= "${__to:date:iso}" and variable="${vars}"
GROUP BY window(timestamp, "${__interval_ms} milliseconds"), variable
ORDER BY time;

At the moment I am not able to use databricks to populate a variable by a query, but hopefully that gets fixed.

Now that databricks has released a dbr-sql client-library for golang, I think it should be quite doable to make a open source and free version of the connector, hopefully someone does that 😉

Re: Grafana + databricks = True?

André — Thu, 07 Jul 2022 10:40:41 GMT

Do you know what functionality is provided by the grafana connector?

Re: Grafana + databricks = True?

wrnjun — Thu, 25 Aug 2022 02:15:45 GMT

Does Grafana have any free ways to connect to Databricks?

Re: Grafana + databricks = True?

Erik — Thu, 25 Aug 2022 07:08:12 GMT

No, there are no free ways yet.

With the newly released databricks-sql-connector for golang it should be quite easy for anyone who known golang to create a grafana backend datasource plugin, so I hope someone does 😉

Re: Grafana + databricks = True?

wrnjun — Thu, 25 Aug 2022 07:16:32 GMT

I hope so, too..

Re: Grafana + databricks = True?

cold_river_22 — Fri, 02 Dec 2022 18:42:20 GMT

There is now an Open-Source Grafana Databricks backend plugin available.

https://github.com/mullerpeter/databricks-grafana

Re: Grafana + databricks = True?

frugson — Sat, 30 Aug 2025 04:03:50 GMT

@Erik wrote:
We have some timeseries in databricks, and we are reading them into powerbi through sql compute endpoints. For timeseries powerbi is ... not optimal. Earlier I have used grafana with various backends, and quite like it, but I cant find any way to connect grafana with databricks. So I have two questions:

Does anyone know if it is possible to use databricks/spark as a datasource for grafana in some way (I have ofcourse done the normal googlesearch)?
Is there any other good way to query, visualize, and explore spark/databricks timeseries?

Grafana doesn’t natively support Databricks, but you can connect via a SQL endpoint using a JDBC or PostgreSQL proxy. Alternatively, tools like Azure Data Explorer or Plotly Dash offer better support for exploring Databricks timeseries.