cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Grafana + databricks = True?

Erik
Valued Contributor II

We have some timeseries in databricks, and we are reading them into powerbi through sql compute endpoints. For timeseries powerbi is ... not optimal. Earlier I have used grafana with various backends, and quite like it, but I cant find any way to connect grafana with databricks. So I have two questions:

  1. Does anyone know if it is possible to use databricks/spark as a datasource for grafana in some way (I have ofcourse done the normal googlesearch)?
  2. Is there any other good way to query, visualize, and explore spark/databricks timeseries?

15 REPLIES 15

Kaniz
Community Manager
Community Manager

Hi @ Erik! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

wrnjun
New Contributor II

I was going to migrate all my data to Databricks, but Databricks does not support Grafana  

André
New Contributor III

Hi,

@Kaniz Fatma​  is there any update on this? I understand this is available: https://databrickslabs.github.io/overwatch/

Although it doesn't have official support which might be an issue. Thank you.

Erik
Valued Contributor II

@André Monteiro​ Unfortunately I think Overwatch uses influx somewhere in there as the backend for those dashboards (if I remember correctly from the presentation @Alex Ott​ had for our company, maybe you can confirm?) But it would definitely be cooler if spark could be the backend;-)

alexott
Valued Contributor II
Valued Contributor II

Overwatch uses Delta to store all the data, so you can use Databricks notebooks or Databricks SQL for generation of dashboards. Influx was mentioned in context of more real-time implementation

Erik
Valued Contributor II

@Alex Ott​ : But the pretty graphs at the bottom of https://databrickslabs.github.io/overwatch/, which looks very much like grafana, they are not backed by spark right (i.e. you guys have not made a spark grafana-datasource as part of overwatch)?

Imagine how cool it would be with a grafana dbr-sql datasource, and you could get real-time info directly from your delta-table into grafana! If we only knew someone proficent in both Go and dbr-sql 😉

alexott
Valued Contributor II
Valued Contributor II

Yes, the screenshots at bottom are for the realtime version, but it requires quite a lot of effort. Regarding Grafana - I recommend to talk with your CSE or SA - maybe it could be prioritized

Erik
Valued Contributor II

So, just to update people, since things have changed a bit! The good news is that Grafana has released a databricks data-connector! The bad news is that its "enterprise only". I have played a bit with it, and it works quite well. Here is an example snippet which fetches a timeseries given the time-bounds given in grafana, and resamples it to the bucket-size grafana proposes:

SELECT window.start as time, avg(value),
FROM
mydb.mytable
WHERE timestamp >= "${__from:date:iso}" and timestamp <= "${__to:date:iso}" and variable="${vars}"
GROUP BY window(timestamp, "${__interval_ms} milliseconds"), variable
ORDER BY time;

At the moment I am not able to use databricks to populate a variable by a query, but hopefully that gets fixed.

Now that databricks has released a dbr-sql client-library for golang, I think it should be quite doable to make a open source and free version of the connector, hopefully someone does that 😉

Kaniz
Community Manager
Community Manager

Hi @Erik Parmann​ , Thank you for your update. We appreciate your attempt to choose the best answer for us.

André
New Contributor III

Do you know what functionality is provided by the grafana connector?

Kaniz
Community Manager
Community Manager

Hi @André Monteiro​, The Databricks integration allows a direct connection to Databricks to query and visualize Databricks data in Grafana.

This interaction provides a SQL editor to format and colour code your SQL statements.

Source

wrnjun
New Contributor II

Does Grafana have any free ways to connect to Databricks?

Erik
Valued Contributor II

No, there are no free ways yet.

With the newly released databricks-sql-connector for golang it should be quite easy for anyone who known golang to create a grafana backend datasource plugin, so I hope someone does 😉

wrnjun
New Contributor II

I hope so, too..

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!