Building RAG without Catalog

[Aminsn]

All the tutorials I have seen about building a RAG within databricks involves using the Catalog option. For some reason my organization doesn't support that option and I can't use the unity catalog to build a RAG. Can I still use other features such as vector search and all the easy to use features that databricks provides without using the catalog? Can someone please refer me to a tutorial of building a RAG with databricks without using the catalog option? 

[raphaelblg · Accepted solution]

Hi @Aminsn, 

Having Unity Catalog enabled is a requirement for vector search. Source: Vector Search - Requirements. If you're referencing this tutorial Creating High Quality RAG Applications with Databricks I think that many of the required features also rely on UC and there's not much you can do to avoid that. I'm not aware of a non-UC tutorial for the scenario you described.

These are the docs on how to enable a workspace for UC, in case you'd like to proceed in this way: https://docs.databricks.com/en/data-governance/unity-catalog/enable-workspaces.html

[Aminsn]

Thank you @raphaelblg , I asked our data engineer, who is our databricks admin, to enable UC and this was his response: 

"It's fairly complex because the Unity Catalog aims to have a single metastore used across all workspaces. We want to implement it at some point due to its numerous benefits. However, doing it correctly requires a significant amount of time."

My data engineering knowledge is very limited, but judging by the instructions provided in the link you shared with me, I don't see any complexity in enabling it. Is there something I am missing regarding the complexities of enabling that option on databricks? 

[raphaelblg]

Hi @Aminsn, 

You don't need to enable all workspaces to Unity Catalog at once, it's possible to enable them by demand. 

We have a very straightforward guide on how to enable UC. The first step is to Create a Unity Catalog metastore and then Enable your workspace for Unity Catalog. It requires some degree of product knowledge but we have docs detailing every step.