Governance for vector search across multi domain unstructured data

Bitrip007 — Thu, 16 Jul 2026 15:23:09 GMT

What is the recommended governance strategy for Databricks Vector Search in an enterprise Unified Data Platform (UDP) where structured and unstructured data from multiple business domains (Sales, Marketing.) is ingested into a common analytics platform?

We are building a multi-domain GenAI/RAG application on Databricks, where data from different business domains is onboarded through a centralized UDP layer. We're evaluating the best approach for Vector Search governance.

Specifically:

Should we create separate Vector Search Indexes for each business domain, or use a single shared index with metadata filtering?
How can Unity Catalog permissions be enforced during vector retrieval to ensure users only access embeddings from authorized domains?
What is the recommended approach for embedding governance, index lifecycle management, and document updates when multiple domains share the same platform?

Re: Governance for vector search across multi domain unstructured data

iyashk-DB — Sun, 19 Jul 2026 09:02:17 GMT

The short answer on index architecture: separate indexes per domain is usually the right call for an enterprise setup, not a single shared index with metadata filters.

Here's why the "one index with filtering" approach is tempting but fragile. Vector Search doesn't support row or column level security. When you query a shared index, the filtering happens at the application layer, which means your RAG layer is responsible for enforcing access, not the platform. That's a governance gap that gets worse as the number of domains grows. If a developer forgets to pass the domain filter, or the filter is misconfigured, users can retrieve embeddings from domains they shouldn't touch. There's no platform-level safety net.

With separate indexes per domain, you get actual access isolation. Each index lives in a catalog/schema scoped to that domain, and Unity Catalog ACLs control who can even call the endpoint. A user without SELECT on the schema can't query the index at all, independent of what the application does. That's enforcement you can audit, not application logic you have to trust.

Index and schema layout for 20+ domains:

Put each domain's indexes in its own schema, ideally under a domain-specific catalog:

catalog: sales_domain
    schema: embeddings
        table: customer_docs_index
        table: product_docs_index

catalog: marketing_domain
    schema: embeddings
        table: campaign_docs_index

Grant the domain team USE CATALOG, USE SCHEMA, and SELECT on their catalog. Cross-domain access is an explicit grant, not a default.

If you do need a shared index (e.g., a cross-domain search experience where the retrieval layer is trusted), use the filter API to scope queries. For standard endpoints:

results = client.similarity_search(
    index_name="main.shared.all_domains_index",
    query_text="...",
    filters={"domain": "sales"}
)

For storage-optimized endpoints, the syntax is SQL-style:

filters="domain = 'sales'"

The important thing to understand is that filters here are applied post-retrieval, not pre-retrieval, on storage-optimized endpoints. Results are overfetched and then trimmed, so you may get fewer results than your num_results parameter specifies if most candidates don't match the filter. This is especially relevant when domain data is sparse in a shared index.

Embedding governance and lifecycle:

For each domain, the source Delta table that backs the index is what you version and govern. A Delta Sync index automatically updates when the source table changes. Document updates flow through the Delta table, UC lineage tracks it, and you can use Delta table history for rollbacks. Treat the index as a derived asset, not a primary one.

For embedding model governance, if you want to control which embedding model is used per domain (to avoid one domain's model choices contaminating another), register your embedding models in UC under the appropriate catalog and grant access per domain.

The unified catalog/schema approach also gives you clean lifecycle management: retiring a domain means dropping its catalog, revoking endpoint access, and the cleanup is isolated.

topic Re: Governance for vector search across multi domain unstructured data in Generative AI

Governance for vector search across multi domain unstructured data

Re: Governance for vector search across multi domain unstructured data