Data security when using Databricks Foundation Model endpoints

[tholers]

Hi Databricks,

My client is considering using Databricks as the solution to their RAG chatbot framework. We are testing on the Databrick's demo tutorial on its own documentation data (pdf files). The tutorial is provided at: 00-RAG-chatbot-Introduction - Databricks

However, we have a concern with data security. If we use the endpoints of the embedding models provided on Databricks (screenshot), will our data stay within our tenant? We are worrying about our data being sent to outside the tenant we have and that will not be accepted.

Thank you.

[Louis_Frolio]

Hey @tholers , here’s how Databricks handles data for the embedding model endpoints and what that means for tenant and data residency requirements.

Short answer

• If you use Databricks-hosted embedding endpoints (for example, GTE-Large or BGE-Large), your requests are processed within the Databricks security perimeter in the same cloud region as your workspace, and are not shared with third-party model providers. Databricks may temporarily retain inputs/outputs for up to 30 days solely to detect and mitigate abuse; these are isolated per customer, stay in your workspace region, and are not used to train any models. Foundation Model APIs are a Designated Service and adhere to Databricks geos and residency boundaries.
• If you instead use external model endpoints (e.g., OpenAI, Anthropic, Gemini), requests are forwarded to those providers under Databricks governance; Databricks is enrolled in OpenAI’s zero data retention, and Anthropic does not have access to your prompts/responses in Databricks’ private tenant on AWS Bedrock, with traffic encrypted in transit.

What “stays within our tenant” means on Databricks

• Foundation Model APIs (pay-per-token or provisioned throughput) are workspace-level APIs managed by Databricks. Requests are authenticated, authorized, stateless, and do not persist context; traffic is encrypted in transit, and models are stored in Databricks-managed private storage with platform-grade security controls. Databricks-hosted endpoints process and retain data per the Model Serving data protection policy noted above (temporary retention up to 30 days for abuse detection, same-region, isolated, not used for training).
• Practically, this means the data is handled within the Databricks platform in your region rather than inside your own cloud subscription/VNet. If your policy requires that no document text ever leaves your tenant’s network, see the options below.

Clarifying the embedding endpoints you mentioned

• Databricks-hosted embedding endpoints (e.g., GTE-Large, BGE-Large) are explicitly documented as “hosted by Databricks Inc. within the Databricks security perimeter,” which implies no third-party provider processing for these embeddings. Data protection and residency rules for these endpoints follow the Model Serving data protection guarantees noted above.

Options if you must keep all text strictly within your own tenant

• Self-host embeddings as a custom model on Mosaic AI Model Serving (provisioned throughput) or run inference on your own compute. You can apply serverless egress controls / network policies to restrict outbound connections from endpoints, and keep persisted artifacts (embeddings, logs) governed in Unity Catalog.
• Avoid external model providers by using only Databricks-hosted foundation models for embedding or by hosting open-source embedding models yourself; the latter keeps the entire inference path within your tenant’s compute and storage boundary (subject to your networking controls).

If you use external model endpoints (for completeness)

• External endpoints unify access to providers like OpenAI/Anthropic/Gemini under Databricks governance (permissions, rate limits, usage tracking, payload logging, AI guardrails). However, your data is sent to the provider under those providers’ data-handling terms; Databricks provides additional guarantees (e.g., OpenAI zero data retention, Databricks private tenant on AWS Bedrock for Anthropic).
• If your client prefers to avoid specific providers, admins can effectively disable those endpoints (e.g., by setting rate limits to zero) or not enable them in the workspace.

Summary guidance

• For most security-sensitive RAG use cases, use Databricks-hosted embedding endpoints to keep processing within the Databricks perimeter and region, with no third-party sharing and no training on your data.
  - If policy requires “never leave our tenant network,” self-host embeddings and control egress/networking; store embeddings in Unity Catalog and govern with AI Gateway features as needed.

Hope this clarifies things for you.

Cheers, Louis.