What is the recommended governance strategy for Databricks Vector Search in an enterprise Unified Data Platform (UDP) where structured and unstructured data from multiple business domains (Sales, Marketing.) is ingested into a common analytics platform?
We are building a multi-domain GenAI/RAG application on Databricks, where data from different business domains is onboarded through a centralized UDP layer. We're evaluating the best approach for Vector Search governance.
Specifically:
- Should we create separate Vector Search Indexes for each business domain, or use a single shared index with metadata filtering?
- How can Unity Catalog permissions be enforced during vector retrieval to ensure users only access embeddings from authorized domains?
- What is the recommended approach for embedding governance, index lifecycle management, and document updates when multiple domains share the same platform?
