07-24-2024 07:07 AM - edited 07-24-2024 07:09 AM
All the tutorials I have seen about building a RAG within databricks involves using the Catalog option. For some reason my organization doesn't support that option and I can't use the unity catalog to build a RAG. Can I still use other features such as vector search and all the easy to use features that databricks provides without using the catalog? Can someone please refer me to a tutorial of building a RAG with databricks without using the catalog option?
07-24-2024 02:38 PM
Hi @Aminsn,
Having Unity Catalog enabled is a requirement for vector search. Source: Vector Search - Requirements. If you're referencing this tutorial Creating High Quality RAG Applications with Databricks I think that many of the required features also rely on UC and there's not much you can do to avoid that. I'm not aware of a non-UC tutorial for the scenario you described.
These are the docs on how to enable a workspace for UC, in case you'd like to proceed in this way: https://docs.databricks.com/en/data-governance/unity-catalog/enable-workspaces.html
07-24-2024 02:38 PM
Hi @Aminsn,
Having Unity Catalog enabled is a requirement for vector search. Source: Vector Search - Requirements. If you're referencing this tutorial Creating High Quality RAG Applications with Databricks I think that many of the required features also rely on UC and there's not much you can do to avoid that. I'm not aware of a non-UC tutorial for the scenario you described.
These are the docs on how to enable a workspace for UC, in case you'd like to proceed in this way: https://docs.databricks.com/en/data-governance/unity-catalog/enable-workspaces.html
07-25-2024 12:40 AM
Thank you @raphaelblg , I asked our data engineer, who is our databricks admin, to enable UC and this was his response:
"It's fairly complex because the Unity Catalog aims to have a single metastore used across all workspaces. We want to implement it at some point due to its numerous benefits. However, doing it correctly requires a significant amount of time."
My data engineering knowledge is very limited, but judging by the instructions provided in the link you shared with me, I don't see any complexity in enabling it. Is there something I am missing regarding the complexities of enabling that option on databricks?
07-25-2024 07:59 AM
Hi @Aminsn,
You don't need to enable all workspaces to Unity Catalog at once, it's possible to enable them by demand.
We have a very straightforward guide on how to enable UC. The first step is to Create a Unity Catalog metastore and then Enable your workspace for Unity Catalog. It requires some degree of product knowledge but we have docs detailing every step.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group