I made an AI assistant for Databricks docs, let me know what you think!

[matte_kapa_bul]

Hello members of the Databricks community!

I built this Ask AI chatbot/widget where I gave a custom LLM access to some of Databricks' docs to help answer technical questions for people using Databricks. I tried it on a couple of questions that resemble the ones asked here or in Stack Overflow, and it answered them within seconds (whenever they related to stuff in the docs, of course). In a nutshell, it helps people interacting with the documentation to get "unstuck" faster, and ideally with less frustrations.

Feel free to try it out here (no login required): https://demo.kapa.ai/widget/databricks

I'd love to get the feedback of the community on this!

P.S. I've read the "Databricks Community Code of Conduct" and I concluded that posting this in here is alright, but if you know better, do let me know! In any case, I hope this is interesting and helpful! 😁

[WiliamRosa]

Hi @matte_kapa_bul , how are you doing?
First of all, congratulations on the initiative. I’ve tried to do something similar myself, and it’s very useful for locating documentation, but it doesn’t end up being very effective in solving some issues reported by community members. I ran some tests with real questions and, for example, there was a case where a service outage was reported, but in my tests this wasn’t the case — I even provided that response in the chat. Cases involving resource parameterization, environment differences, or different cloud providers are still complex for an LLM to accurately evaluate.

[matte_kapa_bul]

Hi @WiliamRosa, I am doing very well, thanks!

First and foremost, awesome that you tried it out and that you trie to do something similar yourself! You are spot on that it's not straightforward to build. We have had a team working on this approx full-time for around 2.5 years to get the accuracy to the level it is today (iterating a lot, hahaha). The trickiest part is when the knowledge isn't in public docs (like in Zendesk tickets, or just in someone's head). Cases like the ones you mentioned are definitely nontrivial to handle, and sometimes it's not possible due to the lack of curated documentation. And that's fine!

On the service outage test you mentioned, I am not totally sure what was asked in the widget (I can't reconnect it to the analytics I have access to), so can I follow-up privately with you? (direct message in here if you are okay with it)