Sharing Vector Search Indexes Across Databricks Workspaces with Terraform

[J_Guenther]

As organizations scale their AI initiatives on Databricks, a common pattern emerges: one team builds and maintains a vector store, while other teams across different workspaces need to query it. Perhaps your ML platform team curates a centralized knowledge base in a producer workspace, and multiple product teams in their own consumer workspaces need semantic search access for their RAG applications.

This article walks through an end-to-end, Infrastructure-as-Code approach to cross-workspace Vector Search sharing using the Databricks Terraform provider. You will learn how to provision a Mosaic AI Vector Search index in a producer workspace, grant least-privilege access to consumer service principals via Unity Catalog, and authenticate cross-workspace queries using OAuth machine-to-machine (M2M) credentials,  all managed declaratively in Terraform.

Prerequisites

• Two Databricks workspaces attached to a shared Unity Catalog metastore (AWS in this guide)
• OpenTofu or Terraform with the Databricks provider configured
• Databricks CLI authenticated with profiles for both workspaces

Architecture Overview

The key insight behind this architecture: Databricks cluster tokens are scoped to a single workspace. A notebook running in the consumer workspace cannot use its ambient credentials to query a Vector Search endpoint in the producer workspace. OAuth M2M solves this — the Databricks SDK exchanges the service principal's client_id and client_secret for a token scoped to the producer workspace, which the Vector Search service accepts.

Setting Up the Foundation

Before provisioning Vector Search resources, you need a shared Unity Catalog metastore and account-level service principals. The metastore provides unified governance across both workspaces, while the service principals serve as the identity for cross-workspace queries.

Shared Metastore

resource "databricks_metastore" "this" {
  provider      = databricks.mws
  name          = "shared-metastore"
  storage_root  = "s3://${aws_s3_bucket.metastore.bucket}/metastore"
  region        = var.aws_region
  force_destroy = true
}
resource "databricks_metastore_assignment" "producer" {
  provider     = databricks.mws
  metastore_id = databricks_metastore.this.id
  workspace_id = var.producer_workspace_id
}

resource "databricks_metastore_assignment" "consumer" {
  provider     = databricks.mws
  metastore_id = databricks_metastore.this.id
  workspace_id = var.consumer_workspace_id
}

Both workspaces must share the same metastore for Unity Catalog grants to apply uniformly.

Account-Level Service Principal

The consumer service principal is created at the account level and then assigned to both workspaces. It needs to exist in the producer workspace to receive endpoint ACLs, and in the consumer workspace to run notebooks there.

resource "databricks_service_principal" "consumer_sp" {
  provider     = databricks.mws
  display_name = "vs-share-consumer-sp"
  active       = true
}

# Assign to both workspaces as USER
resource "databricks_mws_permission_assignment" "consumer_sp_in_producer" {
  provider     = databricks.mws
  workspace_id = var.producer_workspace_id
  principal_id = databricks_service_principal.consumer_sp.id
  permissions  = ["USER"]
}

resource "databricks_mws_permission_assignment" "consumer_sp_in_consumer" {
  provider     = databricks.mws
  workspace_id = var.consumer_workspace_id
  principal_id = databricks_service_principal.consumer_sp.id
  permissions  = ["USER"]
}

Creating the Vector Search Index

With the metastore shared and the service principal assigned to both workspaces, the next phase provisions the Vector Search resources in the producer workspace: an endpoint to serve queries and an index backed by a Delta table.

Endpoint and Index

resource "databricks_vector_search_endpoint" "producer" {
  provider      = databricks.workspace_producer
  name          = "producer-vs-endpoint"
  endpoint_type = "STANDARD"
}

resource "databricks_vector_search_index" "documents" {
  provider      = databricks.workspace_producer
  name          = "${var.catalog_name}.${var.schema_name}.documents_vs_index"
  endpoint_name = databricks_vector_search_endpoint.producer.name
  primary_key   = "id"
  index_type    = "DELTA_SYNC"
  delta_sync_index_spec {
    source_table  = "${var.catalog_name}.${var.schema_name}.documents"
    pipeline_type = "TRIGGERED"
    embedding_source_columns {
      name                          = "content"
      embedding_model_endpoint_name = "databricks-gte-large-en"
    }
  }
}

This creates a Delta Sync index with managed embeddings. Databricks automatically generates vector embeddings for the content column using databricks-gte-large-en — no external embedding service required. The TRIGGERED pipeline type means the index syncs on demand rather than continuously, giving you control over when embedding computation runs.

Source Table Requirements

The source Delta table must have Change Data Feed (CDF) enabled — this is a requirement for all Delta Sync indexes:

spark.sql(f"""
  CREATE TABLE IF NOT EXISTS {catalog}.{schema}.documents (
    id      STRING NOT NULL,
    title   STRING NOT NULL,
    content STRING NOT NULL
  )
  USING DELTA
  TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true')
""")

After populating the table and applying the Terraform configuration, trigger the initial sync:

databricks vector-search indexes sync-index \
  --index-name producer_catalog.vector_data.documents_vs_index \
  --profile one-env-producer

The Permission Model: Two Layers

Cross-workspace Vector Search access requires permissions at two distinct layers. Both must be satisfied for a query to succeed.

Layer 1: Unity Catalog Grants

Unity Catalog controls access to the data itself. Each consumer service principal needs three grants to traverse the catalog hierarchy and read the index. The example below shows two consumer SPs — the pattern scales to any number by adding more grant {} blocks:

resource "databricks_grants" "catalog" {
  provider = databricks.workspace_producer
  catalog  = databricks_catalog.producer.name
  grant {
    principal  = var.consumer_sp_application_id
    privileges = ["USE_CATALOG"]
  }
  grant {
    principal  = var.consumer_sp_readonly_application_id
    privileges = ["USE_CATALOG"]
  }
}

resource "databricks_grants" "schema" {
  provider = databricks.workspace_producer
  schema   = "${databricks_catalog.producer.name}.${databricks_schema.vector_data.name}"
  grant {
    principal  = var.consumer_sp_application_id
    privileges = ["USE_SCHEMA"]
  }
  grant {
    principal  = var.consumer_sp_readonly_application_id
    privileges = ["USE_SCHEMA"]
  }
}

resource "databricks_grants" "vs_index" {
  provider = databricks.workspace_producer
  table    = databricks_vector_search_index.documents.name
  grant {
    principal  = var.consumer_sp_application_id
    privileges = ["SELECT"]
  }
  grant {
    principal  = var.consumer_sp_readonly_application_id
    privileges = ["SELECT"]
  }

}

Important: The databricks_grants resource is authoritative — it replaces all existing grants on the target object. Every principal that needs access must appear as a grant {} block within the same resource. Creating a second databricks_grants resource for the same object will silently overwrite the first, removing its grants.

Layer 2: Endpoint ACLs

The Vector Search endpoint has its own workspace-level access control, managed separately from Unity Catalog:

resource "databricks_permissions" "vs_endpoint" {
  provider = databricks.workspace_producer
  vector_search_endpoint_id = databricks_vector_search_endpoint.producer.endpoint_id
  access_control {
    service_principal_name = var.consumer_sp_application_id
    permission_level       = "CAN_USE"
  }
}

CAN_USE grants query access without management capabilities. The consumer SP can execute similarity searches but cannot modify the endpoint configuration or delete indexes.

Gotcha: The vector_search_endpoint_id field requires the endpoint's UUID (.endpoint_id), not its name (.id). Using .id causes a silent failure — Terraform applies without error, but the permission is never actually set.

Secure Credential Management

For cross-workspace authentication, the consumer notebook needs the service principal's OAuth credentials. Rather than hardcoding secrets or passing them as job parameters, store them in a Databricks secret scope managed by Terraform:

resource "databricks_secret_scope" "vs_share" {
  provider = databricks.workspace_consumer
  name     = "vs-share"
}

resource "databricks_secret" "consumer_sp_client_id" {
  provider     = databricks.workspace_consumer
  scope        = databricks_secret_scope.vs_share.name
  key          = "consumer-sp-client-id"
  string_value = var.consumer_sp_application_id
}

resource "databricks_secret" "consumer_sp_client_secret" {
  provider     = databricks.workspace_consumer
  scope        = databricks_secret_scope.vs_share.name
  key          = "consumer-sp-client-secret"
  string_value = var.consumer_sp_client_secret
}
# Grant the SP READ access to its own secret scope
resource "databricks_secret_acl" "consumer_sp_read" {
  provider   = databricks.workspace_consumer
  scope      = databricks_secret_scope.vs_share.name
  principal  = var.consumer_sp_application_id
  permission = "READ"
}

Each service principal gets its own secret scope with a READ-only ACL. This ensures that even if multiple SPs exist in the workspace, they can only access their own credentials.

Why OAuth M2M over Personal Access Tokens? OAuth M2M credentials are account-level and purpose-built for service-to-service communication. The Databricks SDK handles token lifecycle automatically — tokens are short-lived and refreshed transparently — and you can rotate secrets independently per service principal. PATs, by contrast, are tied to individual users, require manual rotation, and blur the audit trail.

Querying Cross-Workspace: Two Approaches

With permissions and credentials in place, the consumer notebook can query the producer's Vector Search index. Both approaches below start with the same OAuth M2M authentication:

from databricks.sdk import WorkspaceClient
# Load M2M credentials from the consumer workspace's secret scope
CLIENT_ID     = dbutils.secrets.get("vs-share", "consumer-sp-client-id")
CLIENT_SECRET = dbutils.secrets.get("vs-share", "consumer-sp-client-secret")

# Authenticate to the producer workspace via OAuth M2M.
# The SDK performs a client credentials flow against the producer's OIDC
# token endpoint, returning a token scoped to the producer workspace.
producer = WorkspaceClient(
    host=PRODUCER_WORKSPACE_URL,
    client_id=CLIENT_ID,
    client_secret=CLIENT_SECRET,
)

From here, you have two options for issuing queries.

Approach A: Databricks SDK (Recommended)

The SDK's query_index() method provides typed responses, built-in retry logic, and native filter support:

# Basic similarity search
results = producer.vector_search_indexes.query_index(
    index_name=VS_INDEX_NAME,
    query_text="machine learning neural networks",
    columns=["id", "title", "content"],
    num_results=5,
)



for row in results.result.data_array:
    doc_id, title, content, score = row[0], row[1], row[2], row[-1]
    print(f"  [{score:.4f}] {title}")

# Filtered search — exclude a specific document
results = producer.vector_search_indexes.query_index(
    index_name=VS_INDEX_NAME,
    query_text="retrieval augmented generation RAG",
    columns=["id", "title", "content"],
    num_results=3,
    filters_json='{"id NOT": 1}',
)

Approach B: REST API

For custom HTTP clients or languages without a Databricks SDK, call the Vector Search REST API directly:

import requests

auth_headers = {**producer.config.authenticate(), "Content-Type": "application/json"}

resp = requests.post(
    f"{PRODUCER_WORKSPACE_URL}/api/2.0/vector-search/indexes/{VS_INDEX_NAME}/query",
    headers=auth_headers,
    json={
        "query_text": "data governance access control",
        "num_results": 3,
        "columns": ["id", "title", "content"],
    },
)
resp.raise_for_status()
for row in resp.json()["result"]["data_array"]:
    doc_id, title, content, score = row[0], row[1], row[2], row[-1]
    print(f"  [{score:.4f}] {title}")

The authentication is identical in both cases. The SDK approach is preferred for Python workloads; the REST approach is useful when you need full control over the HTTP layer or are working in a language without SDK support.

Running as a Service Principal

In production, you want queries to run under the service principal's identity — not a human user's. Databricks jobs support a run_as block that executes the entire job as a specified service principal:

resource "databricks_job" "consumer_demo" {
  provider = databricks.workspace_consumer
  name     = "vs-share-cross-workspace-query-demo"
  run_as {
    service_principal_name = var.consumer_sp_application_id
  }
  task {
    task_key = "cross_workspace_query"
    notebook_task {
      notebook_path = databricks_notebook.consumer_demo.path
      source        = "WORKSPACE"
      base_parameters = {
        producer_workspace_url = var.producer_workspace_url
        vs_index_name          = "producer_catalog.vector_data.documents_vs_index"
      }
    }
    # Omitting the cluster spec causes the task to run on serverless compute.
  }
}

For run_as to work, the Terraform runner (or whoever creates the job) must hold the servicePrincipal.user role on the target SP. This is an account-level permission managed via access control rule sets:

resource "databricks_access_control_rule_set" "consumer_sp_users" {
  provider = databricks.mws
  name     = "accounts/${var.databricks_account_id}/servicePrincipals/${databricks_service_principal.consumer_sp.application_id}/ruleSets/default"
  grant_rules {
    principals = ["users/${data.databricks_current_user.tf.user_name}"]
    role       = "roles/servicePrincipal.user"
  }
}

Running jobs as service principals provides a clean audit trail — every query is attributed to the SP, not a human user — and ensures that no personal credentials are embedded in production workloads.

Least-Privilege Security Checklist

The consumer service principal receives exactly the permissions it needs and nothing more:

Permission	Scope	Purpose
`USE_CATALOG`	Producer catalog	Traverse the catalog hierarchy
`USE_SCHEMA`	Producer schema	Traverse the schema
`SELECT`	VS index	Read-only data access (no writes or deletes)
`CAN_USE`	VS endpoint	Query access (no management)
`READ`	Own secret scope	Access its own M2M credentials

Notably absent: ALL_PRIVILEGES, MODIFY, CREATE TABLE, CAN_MANAGE. The SP cannot alter the index, modify the endpoint configuration, create new objects, or access other service principals' credentials. This aligns with data mesh principles where the producer team retains full control over the data asset while granting precise, auditable read access to consumers.

Conclusion

By combining Unity Catalog's cross-workspace governance, OAuth M2M authentication, and the Databricks Terraform provider, you can build a secure, repeatable pattern for sharing Vector Search indexes across workspace boundaries. The key components:

• Unity Catalog provides a unified permission model across workspaces
• OAuth M2M enables service-to-service authentication without personal credentials
• Terraform makes the entire setup declarative, version-controlled, and reproducible

From here, you can extend the pattern in several directions: switch the index to pipeline_type = "CONTINUOUS" for near-real-time sync, onboard additional consumer workspaces by adding new service principals and grant {} blocks, or wire secret rotation through your organization's vault with Terraform's lifecycle management. Because everything is declared in code, each new consumer inherits the same least-privilege security model automatically.