Performance at Scale

Where MCP servers actually slow down

A server that feels instant with one user calling it occasionally can feel sluggish the moment real traffic shows up. Almost all of that gap comes down to four things: repeating work that didn't need repeating, opening a new connection for every request, handling items one at a time that could be handled together, and returning more data than the model actually needed to see.

Caching repeated lookups

If the same query runs often and the underlying data doesn't change every second, cache it:

from datetime import datetime, timedelta

class TicketCache:
    def __init__(self, ttl_seconds: int = 60):
        self.ttl = timedelta(seconds=ttl_seconds)
        self.store: dict[str, tuple] = {}

    async def get_or_fetch(self, key: str, fetch):
        cached = self.store.get(key)
        if cached and datetime.utcnow() - cached[1] < self.ttl:
            return cached[0]
        value = await fetch()
        self.store[key] = (value, datetime.utcnow())
        return value

A short TTL, even 60 seconds, absorbs a surprising amount of repeated load without ever serving data stale enough to matter. Adjust it per resource: an item's price probably needs a shorter TTL than its description.

Reusing connections

Opening a fresh connection to a downstream service on every single call adds latency that has nothing to do with the actual work being done:

import aiohttp

class SharedSession:
    _session: aiohttp.ClientSession | None = None

    @classmethod
    async def get(cls) -> aiohttp.ClientSession:
        if cls._session is None or cls._session.closed:
            cls._session = aiohttp.ClientSession(
                timeout=aiohttp.ClientTimeout(total=15)
            )
        return cls._session

One shared, reused session for a given downstream service avoids the connection setup cost on every call, which is often a larger share of total latency than the request itself.

Batching instead of looping

A tool that updates ten records with ten separate calls pays the overhead ten times. One batched call pays it once:

@mcp.tool()
async def close_tickets(ticket_ids: list[str]) -> dict:
    """Close multiple tickets in a single database round trip"""
    result = await db.execute(
        "UPDATE tickets SET status = 'closed' WHERE id = ANY($1)",
        ticket_ids,
    )
    return {"closed": result.rowcount}

If a tool's natural use case is "do this to several things at once," design it to accept a list from the start, rather than relying on the model to call a single-item version in a loop.

Returning only what's needed

The model has to read whatever a tool returns, and every unnecessary field costs tokens without adding value:

def summarize_for_response(ticket: dict) -> dict:
    """Return the fields a response actually needs, not the full record"""
    return {
        "id": ticket["id"],
        "status": ticket["status"],
        "summary": ticket["summary"],
    }

A tool that returns a full internal record when the model only needed the status field is quietly making every response more expensive and slower to reason over than it needs to be. Default to a lean response, and add an explicit parameter like detail_level for the cases that genuinely need more.