12-16-2021 11:23 AM
We have some timeseries in databricks, and we are reading them into powerbi through sql compute endpoints. For timeseries powerbi is ... not optimal. Earlier I have used grafana with various backends, and quite like it, but I cant find any way to connect grafana with databricks. So I have two questions:
12-16-2021 11:44 AM
Hi @ Erik! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.
12-21-2021 03:31 AM
I was going to migrate all my data to Databricks, but Databricks does not support Grafana
03-03-2022 01:40 PM
Hi,
@Kaniz Fatma is there any update on this? I understand this is available: https://databrickslabs.github.io/overwatch/
Although it doesn't have official support which might be an issue. Thank you.
03-07-2022 03:18 AM
@André Monteiro Unfortunately I think Overwatch uses influx somewhere in there as the backend for those dashboards (if I remember correctly from the presentation @Alex Ott had for our company, maybe you can confirm?) But it would definitely be cooler if spark could be the backend;-)
03-07-2022 03:46 AM
Overwatch uses Delta to store all the data, so you can use Databricks notebooks or Databricks SQL for generation of dashboards. Influx was mentioned in context of more real-time implementation
03-07-2022 04:32 AM
@Alex Ott : But the pretty graphs at the bottom of https://databrickslabs.github.io/overwatch/, which looks very much like grafana, they are not backed by spark right (i.e. you guys have not made a spark grafana-datasource as part of overwatch)?
Imagine how cool it would be with a grafana dbr-sql datasource, and you could get real-time info directly from your delta-table into grafana! If we only knew someone proficent in both Go and dbr-sql 😉
03-07-2022 05:02 AM
Yes, the screenshots at bottom are for the realtime version, but it requires quite a lot of effort. Regarding Grafana - I recommend to talk with your CSE or SA - maybe it could be prioritized
07-05-2022 01:34 AM
So, just to update people, since things have changed a bit! The good news is that Grafana has released a databricks data-connector! The bad news is that its "enterprise only". I have played a bit with it, and it works quite well. Here is an example snippet which fetches a timeseries given the time-bounds given in grafana, and resamples it to the bucket-size grafana proposes:
SELECT window.start as time, avg(value),
FROM
mydb.mytable
WHERE timestamp >= "${__from:date:iso}" and timestamp <= "${__to:date:iso}" and variable="${vars}"
GROUP BY window(timestamp, "${__interval_ms} milliseconds"), variable
ORDER BY time;
At the moment I am not able to use databricks to populate a variable by a query, but hopefully that gets fixed.
Now that databricks has released a dbr-sql client-library for golang, I think it should be quite doable to make a open source and free version of the connector, hopefully someone does that 😉
07-05-2022 03:14 AM
Hi @Erik Parmann , Thank you for your update. We appreciate your attempt to choose the best answer for us.
07-07-2022 03:40 AM
Do you know what functionality is provided by the grafana connector?
07-07-2022 04:01 AM
Hi @André Monteiro, The Databricks integration allows a direct connection to Databricks to query and visualize Databricks data in Grafana.
This interaction provides a SQL editor to format and colour code your SQL statements.
08-24-2022 07:15 PM
Does Grafana have any free ways to connect to Databricks?
08-25-2022 12:08 AM
No, there are no free ways yet.
With the newly released databricks-sql-connector for golang it should be quite easy for anyone who known golang to create a grafana backend datasource plugin, so I hope someone does 😉
08-25-2022 12:16 AM
I hope so, too..
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group