Security considerations and model customization options for AgentBricks AI Agents and LLM Judges

[shivamrai162]

Hello Databricks Team,

We are currently evaluating AgentBricks AI Agents (for example, Knowledge Assistant and Multi-Agent Supervisor) and would like to better understand the security and model customization aspects.

1. Data security & privacy

   • What data (user prompts, retrieved context, tool outputs, intermediate agent reasoning, etc.) is transmitted or persisted when using AgentBricks AI Agents?

   • Is this data logged, stored, or retained by Databricks services, and if so, for how long?

   • How does Databricks ensure data isolation and confidentiality, especially when agents interact with external tools or services?

2. Using custom or non-default models with AgentBricks

   • Is it possible to use a custom locally hosted or self-managed model (for example, a model downloaded and hosted outside Databricks) as the backing LLM for AgentBricks AI Agents?

   • If this is not supported directly, what are the recommended approaches to use a model other than the default Databricks-provided models (for example, via external model endpoints, API-based integration, or other supported mechanisms)?

3. LLM Judges / evaluation models

   • Do the same constraints and options apply to LLM Judges used for evaluating agent or model responses?

   • Can LLM Judges be configured to use a non-default or externally hosted model, and are there any specific security or compliance considerations for this setup?

Any guidance, documentation references, or best-practice recommendations would be greatly appreciated.

Thank you in advance for your support.

[pavannaidu]

@shivamrai162 
1. Data security & privacy

• Agent Bricks uses default storage to store temporary data transformations, model checkpoints, and internal metadata that power each agent. On agent deletion, all data associated with the agent is removed from default storage.
• As a Databricks Designated Service, Agent Bricks uses Databricks Geos to manage data residency when processing customer content.

2. Using custom or non-default models with AgentBricks

• Agent Bricks is a family of products that includes some managed agents (Knowledge Assistant and Supervisor Agent), and the ability to build and deploy custom agents with code.
• Custom agents with code can use any model supported in AI Gateway or hosted on Databricks. I would encourage you to explore Agent Framework if the goal is to build agents using other Databricks-hosted models or externally hosted models through the AI gateway.

3. LLM Judges/evaluation models:

• You can always add custom scorers/judges to any MLflow experiment. Please refer to: Information about the models powering LLM judges
• Which externally hosted models are you considering for LLM judges and scoring? Most models are hosted in Databricks. Refer to: Model requirements for trace-based judges

Agent Bricks is evolving quickly, and things might change, but for now, I hope this helps.