Grafana + databricks = True?

[Erik]

We have some timeseries in databricks, and we are reading them into powerbi through sql compute endpoints. For timeseries powerbi is ... not optimal. Earlier I have used grafana with various backends, and quite like it, but I cant find any way to connect grafana with databricks. So I have two questions:

1. Does anyone know if it is possible to use databricks/spark as a datasource for grafana in some way (I have ofcourse done the normal googlesearch)?
2. Is there any other good way to query, visualize, and explore spark/databricks timeseries?

[Kaniz]

Hi @ Erik! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[wrnjun]

I was going to migrate all my data to Databricks, but Databricks does not support Grafana  

[André]

Hi,

@Kaniz Fatma​  is there any update on this? I understand this is available: https://databrickslabs.github.io/overwatch/

Although it doesn't have official support which might be an issue. Thank you.

[Erik]

@André Monteiro​ Unfortunately I think Overwatch uses influx somewhere in there as the backend for those dashboards (if I remember correctly from the presentation @Alex Ott​ had for our company, maybe you can confirm?) But it would definitely be cooler if spark could be the backend;-)

[alexott]

Overwatch uses Delta to store all the data, so you can use Databricks notebooks or Databricks SQL for generation of dashboards. Influx was mentioned in context of more real-time implementation

[Erik]

@Alex Ott​ : But the pretty graphs at the bottom of https://databrickslabs.github.io/overwatch/, which looks very much like grafana, they are not backed by spark right (i.e. you guys have not made a spark grafana-datasource as part of overwatch)?

Imagine how cool it would be with a grafana dbr-sql datasource, and you could get real-time info directly from your delta-table into grafana! If we only knew someone proficent in both Go and dbr-sql 😉

[alexott]

Yes, the screenshots at bottom are for the realtime version, but it requires quite a lot of effort. Regarding Grafana - I recommend to talk with your CSE or SA - maybe it could be prioritized

[Erik]

So, just to update people, since things have changed a bit! The good news is that Grafana has released a databricks data-connector! The bad news is that its "enterprise only". I have played a bit with it, and it works quite well. Here is an example snippet which fetches a timeseries given the time-bounds given in grafana, and resamples it to the bucket-size grafana proposes:

SELECT window.start as time, avg(value),
FROM
mydb.mytable
WHERE timestamp >= "${__from:date:iso}" and timestamp <= "${__to:date:iso}" and variable="${vars}"
GROUP BY window(timestamp, "${__interval_ms} milliseconds"), variable
ORDER BY time;

At the moment I am not able to use databricks to populate a variable by a query, but hopefully that gets fixed.

Now that databricks has released a dbr-sql client-library for golang, I think it should be quite doable to make a open source and free version of the connector, hopefully someone does that 😉

[Kaniz]

Hi @Erik Parmann​ , Thank you for your update. We appreciate your attempt to choose the best answer for us.

[André]

Do you know what functionality is provided by the grafana connector?

[Kaniz]

Hi @André Monteiro​, The Databricks integration allows a direct connection to Databricks to query and visualize Databricks data in Grafana.

This interaction provides a SQL editor to format and colour code your SQL statements.

Source

[wrnjun]

Does Grafana have any free ways to connect to Databricks?

[Erik]

No, there are no free ways yet.

With the newly released databricks-sql-connector for golang it should be quite easy for anyone who known golang to create a grafana backend datasource plugin, so I hope someone does 😉

[wrnjun]

I hope so, too..