Hopefully this helps...

You can securely connect your external AI agent to a Model Context Protocol (MCP) server and Unity Catalog while maintaining strong control over authentication and resource management. The method depends on whether MCP is outside or inside Databricks. Below are best practices and details for secure endpoint creation and authentication.

MCP Outside Databricks

When MCP is outside Databricks and needs to access Unity Catalog data, REST API calls will be used. Secure, code-based authentication (no browser) is achievable using these methods:

• Personal Access Token (PAT) Authentication:
  You can generate a Databricks PAT and include it in your REST API request headers using an Authorization token. This is suitable for automation and code-only flows, without browser involvement.

  • Generate PAT in Databricks: Go to User Settings > Access Tokens.

  • Use header:

     

    text

    Authorization: Bearer <token>

  • Store the PAT securely (environment variable, secret manager).

• Service Principal (Workspace-Managed Identity):
  For production and enterprise setups, use a service principal registered with Databricks.

  • Authenticate via client credentials (client ID/secret or certificate) using OAuth2 flows from your MCP or agent’s code.

  • Obtain workspace and catalog access via API, using the principal’s scopes and roles.

• Securing the Endpoint:

  • Host your MCP server on a secure cloud VM or service (ensure HTTPS/TLS).

  • Require authentication for access and provide only secure REST API endpoints.

  • Store credentials (tokens/secrets) outside codebase—preferably with a secret management service.

Secure Endpoints in Databricks

When deploying MCP inside Databricks, you can use Databricks native security and authentication mechanisms:

• Databricks REST Endpoints:

  • You can create WebHDFS or standard REST endpoints in Databricks, protected by workspace authentication.

  • Set up an endpoint using Databricks Jobs, Delta Live Tables, or MLflow model serving.

  • Secure using Bearer tokens (PATs) or service principals, as above.

• Unity Catalog Access Control:

  • Assign workspace, schema, and table permissions to users, groups, or service principals in the Unity Catalog.

  • Only entities with appropriate permissions (via access control lists/policies) can query data.

MCP Inside Databricks: Accessing via Python

• Native Access to Unity Catalog:

  • If MCP is running inside a Databricks workspace (e.g., as a notebook, job, or managed MLflow endpoint), it can access Unity Catalog directly using PySpark, Databricks SQL, or REST API.

  • Authorization is seamless if the code runs under a user/service principal with catalog permissions.

  • Best practice: assign least-privileged roles, and audit usage.

• No Explicit Authorization Required:

  • When running inside Databricks with correct role/ID, explicit extra authentication steps are not needed—Databricks manages session tokens.

  • Access is managed by Databricks’ authentication context, so spark.read.table("catalog.schema.table") will work if permissions are set.

Summary Table: Authentication Approaches

Key Recommendations

• Always use HTTPS for all endpoints.

• Prefer service principals and managed identities for scalable, secure automation.

• Store tokens and secrets securely, using environment variables or cloud secret managers.

• Set fine-grained Unity Catalog permissions to limit data access to only what’s needed.

• For MCP inside Databricks, leverage PySpark/DataFrame APIs for direct access, with minimal authentication setup required.

If you need code samples or steps for endpoint setup or OAuth2 authorization, please specify which platform (Azure, AWS, GCP) and Databricks environment you’re using.

For technical implementation details, including authorization flows and secure endpoint creation inside/outside Databricks, review the official Databricks documentation and your cloud provider's security guidelines.