Step-by-Step Guide to Building Custom MCP Server on Databricks

[gopínath]

Introduction

One of the biggest challenges with LLMs is bridging the gap between static knowledge and real-world actions. MCP solves this by providing a standard way for models to connect with external tools and data sources.

The Model Context Protocol (MCP) is an open standard that allows Large Language Models (LLMs) to connect with external tools, data sources, and APIs in a structured way. Instead of relying only on what the model knows, MCP gives the LLM controlled access to real-time information and custom functionality.

In this article, we’ll walk through a step-by-step guide to build a custom MCP server and deploy it as a Databricks app. Once deployed, we’ll test this MCP server in the Databricks playground by attaching it as a tool to the LLM model.

For demonstration, we’ll build simple tools—one to fetch NSE stock market data and another to show how an agent can interact with Databricks (e.g., creating tables or running SQL queries). Similarly, multiple other tools can be developed to seamlessly integrate with Databricks, such as tools for workflow creation and more.

Note: The tools and resources provided related to stock market data are intended solely for educational purposes and should not be interpreted as financial or investment advice.

We’ll cover the following steps:

• Building the MCP server on a local machine
• Deploying the MCP server as a Databricks app
• Testing the MCP server in the Databricks playground

Prerequisites

1. Databricks workspace with the "Managed MCP Server (Beta)" feature enabled in the Preview settings. (This will be turned on by default once it is GA / Public Preview)
   
2. Databricks CLI installed locally
3. Personal access token(PAT)

 

Build MCP Server Locally

1. Clone the Databricks Labs MCP repo which provides a predefined template for building MCP servers

   git clone https://github.com/databrickslabs/mcp.git

2. The Python file mcp/examples/custom-server/src/custom_server/app.py contains the MCP server code. It uses FastMCP to run the server. To add tools, simply define a Python function and decorate it with @mcp.tool. The template already includes a basic example for addition.
   
3. Let’s extend it by adding a few more tools: 
   • Tool 1: get_stock_info
     Uses the yfinance package to fetch live stock details for a given NSE symbol.
   • Tool 2: run_query
     Uses the databricks-sql-connector to execute SQL queries on a Databricks warehouse.
     

Although these tools or Databricks resources can be invoked directly via API or SDK in agent code, wrapping them in an MCP server as Databricks apps provides additional benefits. MCP standardizes the interface, making tools automatically discoverable across downstream agentic frameworks, ensures consistent responses, and eliminates the need for each team to write and maintain separate code. 

Note: In addition to MCP, Databricks also offers the option to create tools using Unity Catalog function tools and Agent Code tools. For more information, please refer to https://docs.databricks.com/aws/en/generative-ai/agent-framework/agent-tool#choose-your-tool-approac...

Adding the code snippet for each tools

import yfinance as yf
import requests
import pandas as pd
from io import StringIO
import databricks.sql
import os
from databricks.sdk.core import Config
from mcp.server.fastmcp import FastMCP

# create MCP server
mcp = FastMCP("custom MCP server on databricks apps")

# Tool 1
@mcp.tool(
    name = "get_stock_info",
    description = "Fetch the stock information of the given symbol"
)
def get_stock_info(symbol: str) -> str | None:
    """Fetch market cap, current price, and 52-week range for NSE stock."""
    try:
        info = yf.Ticker(f"{symbol}.NS").fast_info
        return (
            f"\nDetails for {symbol}.NS:\n"
            f"market_cap = {info['marketCap']}\n"
            f"current_price = {info['last_price']} INR\n"
            f"year_low = {info['year_low']} INR\n"
            f"year_high = {info['year_high']} INR\n"
        )
    except Exception as e:
        print(f"Error fetching {symbol}: {e}")
        return None

# Tool 2
@mcp.tool(
    name="run_query_on_databricks",
    description="Gets the SQL query and execute the query on databricks SQL warehouse"
)
def run_query_on_databricks(sql_query: str):
    """
    Executes a SQL query on a Databricks SQL Warehouse and returns the results as a pandas DataFrame.
    """
    cfg = Config()
    warehouse_id = os.getenv("WAREHOUSE_ID")
    host = cfg.host
    http_path = f"/sql/1.0/warehouses/{warehouse_id}"
    if not all([host, http_path]):
        raise ValueError("Missing required Databricks connection environment variables.")

    with databricks.sql.connect(server_hostname=host, http_path=http_path, credentials_provider=lambda: cfg.authenticate) as conn, conn.cursor() as cur:
        cur.execute(sql_query)
        rows = cur.fetchall()
        return pd.DataFrame(rows, columns=[c[0] for c in cur.description])

 4. Update environment variables in app.yaml

command: ["uv", "run", "custom-server"]
env:
  - name: 'WAREHOUSE_ID'
    value: '<warehouse-id>'

5. Add required Python libraries in requirements.txt

uv
yfinance
pandas
databricks-sql-connector
python-dotenv

With the above steps, we’ve successfully set up the MCP server in our local IDE. Now its time to do the deployment.

 

Deploy the MCP Server as a Databricks App

Deploying the MCP server as a Databricks app can be done in just a few steps:

1. Install the Databricks CLI on your local machine and authenticate to your workspace using OAuth.
   Refer to the Databricks CLI installation guide and complete the required Authentication steps
   

   databricks configure # you will be asked to enter host url and the PAT token
   databricks auth login --host <workspace-url>​

2. Create the Databricks app. The app name should be prefixed with `mcp-`
   

   databricks apps create mcp-<app_name>

   
   
3. Upload the source code to the Databricks workspace and deploy the app.
   

   DATABRICKS_USERNAME=$(databricks current-user me | jq -r .userName)
   databricks sync . "/Users/$DATABRICKS_USERNAME/mcp-custom-server"
   databricks apps deploy mcp-<app_name> --source-code-path "/Workspace/Users/$DATABRICKS_USERNAME/mcp-custom-server/custom-server"

   
   mcp deploy
   
4. Service Principal Permission
   Ensure the service principal has the necessary permissions to run the query. You can find the service principal under the App Authorization section in the deployed Databricks Apps page and grant the required Unity Catalog access.
   

   GRANT ALL ON CATALOG <catalog_name> TO <service_principal>

   
   app service principal
   Now the MCP server is successfully deployed as Databricks App.
   

 

Use the MCP Server in Databricks Playground

1. With the custom MCP server deployed, we can now test it in the Databricks Playground. Choose the LLM model and add the tool by selecting the MCP server as shown below
   add mcp tool with llm
2. Add a System Prompt to guide the Agent on the tasks to perform.
   system prompt
3. Try asking the Agent to fetch the stock details of NSE listed companies. The LLM will internally call the tool get_stock_info, which we defined in the MCP server.
4. Ask the Agent to capture the Top 5 companies by market cap and store the results in a Unity Catalog Delta table. Internally LLM follows the below actions 
   • With its own knowledge, it gets the top 5 companies.
   • For each company, it calculates stock information such as market cap by calling get_stock_info tool.
   • Using the tool run_query_on_databricks, it creates a Delta table and inserts the top 5 company detailsllm response
5.  Lets query this table in Databricks SQL Editor to verify the results.
   result delta table
   
   

This custom MCP server can be called with LangGraph or any agent flow locally or in databricks notebooks.
Refer to langgraph-mcp-tool-calling-agent - Databricks

 

Conclusion

Through this walkthrough, we saw how easy it is to extend Databricks with a custom MCP server — from deployment as an App to testing it live in the Playground. 

Databricks is making it simple to bring AI Agents and real-world data together in one place. Instead of juggling multiple systems, you can now empower your LLMs to fetch external insights, run SQL on your lakehouse, and interact with enterprise data securely — all within the Databricks platform.

 

References

https://docs.databricks.com/aws/en/generative-ai/mcp/custom-mc