Grafana + databricks = True?

[Erik]

We have some timeseries in databricks, and we are reading them into powerbi through sql compute endpoints. For timeseries powerbi is ... not optimal. Earlier I have used grafana with various backends, and quite like it, but I cant find any way to connect grafana with databricks. So I have two questions:

1. Does anyone know if it is possible to use databricks/spark as a datasource for grafana in some way (I have ofcourse done the normal googlesearch)?
2. Is there any other good way to query, visualize, and explore spark/databricks timeseries?

[wrnjun]

I was going to migrate all my data to Databricks, but Databricks does not support Grafana  

[André]

Hi,

@Kaniz Fatma​  is there any update on this? I understand this is available: https://databrickslabs.github.io/overwatch/

Although it doesn't have official support which might be an issue. Thank you.

[Erik]

@André Monteiro​ Unfortunately I think Overwatch uses influx somewhere in there as the backend for those dashboards (if I remember correctly from the presentation @Alex Ott​ had for our company, maybe you can confirm?) But it would definitely be cooler if spark could be the backend;-)

[alexott]

Overwatch uses Delta to store all the data, so you can use Databricks notebooks or Databricks SQL for generation of dashboards. Influx was mentioned in context of more real-time implementation

[Erik]

@Alex Ott​ : But the pretty graphs at the bottom of https://databrickslabs.github.io/overwatch/, which looks very much like grafana, they are not backed by spark right (i.e. you guys have not made a spark grafana-datasource as part of overwatch)?

Imagine how cool it would be with a grafana dbr-sql datasource, and you could get real-time info directly from your delta-table into grafana! If we only knew someone proficent in both Go and dbr-sql 😉

[alexott]

Yes, the screenshots at bottom are for the realtime version, but it requires quite a lot of effort. Regarding Grafana - I recommend to talk with your CSE or SA - maybe it could be prioritized

[Erik]

So, just to update people, since things have changed a bit! The good news is that Grafana has released a databricks data-connector! The bad news is that its "enterprise only". I have played a bit with it, and it works quite well. Here is an example snippet which fetches a timeseries given the time-bounds given in grafana, and resamples it to the bucket-size grafana proposes:

SELECT window.start as time, avg(value),
FROM
mydb.mytable
WHERE timestamp >= "${__from:date:iso}" and timestamp <= "${__to:date:iso}" and variable="${vars}"
GROUP BY window(timestamp, "${__interval_ms} milliseconds"), variable
ORDER BY time;

At the moment I am not able to use databricks to populate a variable by a query, but hopefully that gets fixed.

Now that databricks has released a dbr-sql client-library for golang, I think it should be quite doable to make a open source and free version of the connector, hopefully someone does that 😉

[André]

Do you know what functionality is provided by the grafana connector?

[wrnjun]

Does Grafana have any free ways to connect to Databricks?

[Erik]

No, there are no free ways yet.

With the newly released databricks-sql-connector for golang it should be quite easy for anyone who known golang to create a grafana backend datasource plugin, so I hope someone does 😉

[wrnjun]

I hope so, too..

[cold_river_22]

There is now an Open-Source Grafana Databricks backend plugin available.

https://github.com/mullerpeter/databricks-grafana