topic Re: Building RAG without Catalog in Generative AI

Building RAG without Catalog

Aminsn — Wed, 24 Jul 2024 14:09:48 GMT

All the tutorials I have seen about building a RAG within databricks involves using the Catalog option. For some reason my organization doesn't support that option and I can't use the unity catalog to build a RAG. Can I still use other features such as vector search and all the easy to use features that databricks provides without using the catalog? Can someone please refer me to a tutorial of building a RAG with databricks without using the catalog option?

Re: Building RAG without Catalog

raphaelblg — Wed, 24 Jul 2024 21:38:09 GMT

Hi @Aminsn,

Having Unity Catalog enabled is a requirement for vector search. Source: Vector Search - Requirements. If you're referencing this tutorial Creating High Quality RAG Applications with Databricks I think that many of the required features also rely on UC and there's not much you can do to avoid that. I'm not aware of a non-UC tutorial for the scenario you described.

These are the docs on how to enable a workspace for UC, in case you'd like to proceed in this way: https://docs.databricks.com/en/data-governance/unity-catalog/enable-workspaces.html

Re: Building RAG without Catalog

Aminsn — Thu, 25 Jul 2024 07:40:51 GMT

Thank you @raphaelblg , I asked our data engineer, who is our databricks admin, to enable UC and this was his response:

"It's fairly complex because the Unity Catalog aims to have a single metastore used across all workspaces. We want to implement it at some point due to its numerous benefits. However, doing it correctly requires a significant amount of time."

My data engineering knowledge is very limited, but judging by the instructions provided in the link you shared with me, I don't see any complexity in enabling it. Is there something I am missing regarding the complexities of enabling that option on databricks?

Re: Building RAG without Catalog

raphaelblg — Thu, 25 Jul 2024 14:59:59 GMT

Hi @Aminsn,

You don't need to enable all workspaces to Unity Catalog at once, it's possible to enable them by demand.

We have a very straightforward guide on how to enable UC. The first step is to Create a Unity Catalog metastore and then Enable your workspace for Unity Catalog. It requires some degree of product knowledge but we have docs detailing every step.