AI/BI Genie - Real world experience?

[samueldaly]

I have been tasked with creating a NLP interface for management to our internal datalake that has numerous joined tables around customer information. I have tried the hello-world llamaindex, langchain sql/pandas agents with limited success (too many hallucinations).

Genie sounds promising but keen to hear real world feedback if any exists. 

How did you implement?

How did you benchmark?

What accuracy are you getting?

How are you managing user experience where it makes mistakes?

Txs
Sam

[mitchellg-db]

Hi there Sam,

I work for Databricks. The pace of innovation with AI/BI is wild (fast). Databricks itself just migrated off a leading BI platform and now exclusively runs its business on AI/BI (Blog). We also have customer testimonials on our website, which you can check out (Example). I've been pleasantly surprised by how well Genie performs.

To address your specific questions, we have recently published a guide on achieving the best results with Genie. If you treat it as a set it and forget it BI tool, you will be disappointed, but if you treat it as a living/curated asset, the impact can be immense. We also have great documentation on our benchmarking, feedback, and monitoring capabilities.

You can also reach out to your Databricks account team, and they might be able to set you up with a more specific reference conversation with an active customer.