Remote MCP Server & AI Agent Gateway: Machine-to-Machine Knowledge Retrieval

How we transformed herrington.ai into a first-class programmatic tool server for external AI agents using a public stateless Streamable HTTP MCP architecture.

🔍 The Problem Space: The Web Was Built for Eyes, Not Agents

For three decades, the World Wide Web has been engineered around human ocular consumption: cascading stylesheets, responsive viewports, typography hierarchies, and clickable DOM trees.

Over the past two years, as autonomous AI coding harnesses (Cursor, Windsurf, Claude Code) and autonomous research agents (Muse, Instinct, Perplexity Computer) emerged, they faced an insurmountable friction barrier: the web is actively hostile to non-human intelligence.

When an external AI agent wants to consult Daniel Herrington's body of research on ML Productivity Goodput in LLM Training (arXiv:2502.06982), extract his exact perspective on "The End of Vibe Coding", or review an architectural case study, it was previously forced into one of two inefficient degradation paths:

1. Headless Browser Driving: Spawning a full Chromium instance via Playwright or Puppeteer, executing client-side hydration, and driving the DOM like a human puppet just to click the floating dh.sh terminal and parse characters out of a virtual xterm.js canvas.
2. Brute-Force Scraping: Fetching HTML pages, choking on layout markup, navigation headers, SVG icons, and UTM tags, burning thousands of LLM context tokens on structural noise before reaching the actual engineering insight.

Even with standard discovery mechanisms like /llms.txt, modern reasoning agents require active bidirectional querying—they need the ability to search an index, retrieve specific document chunks on demand, inspect project schemas, and query specialized domain models as first-class function calls.

We set out to eliminate this friction entirely by turning herrington.ai into an open, machine-callable Model Context Protocol (MCP) server accessible at https://herrington.ai/mcp.

🧭 The Critical User Journeys (CUJs)

Following the CUJ-Driven Development methodology, we mapped the machine-to-machine experience across three distinct journeys:

1. Autonomous AI Agent Discovery (Muse & Instinct)

"I want autonomous AI agents like Muse and Instinct to query Daniel's technical writings, architecture projects, and simulated engineering persona via standard Model Context Protocol (MCP) tool calls while avoiding headless browser emulation, DOM scraping, and brittle prompt parsing so that agents can reason over verified knowledge and synthesize citations in milliseconds with zero operational overhead."

External agents require deterministic JSON-RPC 2.0 signatures. They must be able to perform initialize to negotiate capabilities, invoke tools/list to inspect available interfaces, and call search_essays or get_essay with precise arguments, receiving structured outputs that fit directly into their reasoning context.

2. Developer & IDE Integration (Cursor, Claude Desktop, Windsurf)

"I want to configure https://herrington.ai/mcp directly as a remote HTTP MCP server in my Cursor, Claude Desktop, or Windsurf config while avoiding session authorization failures and local proxy containers so that Daniel's engineering frameworks, ML Goodput papers, and codebases are available as native tools in my coding workflow."

Engineers writing software should not need to clone git repositories or run Docker sidecars just to reference Daniel's systems blueprints. By adding a single URL to claude_desktop_config.json or .cursor/mcp.json, the entire knowledge graph becomes an ambient pair-programming tool.

3. Serverless Reliability & Cost Boundary (Cloud Run & Gemini Gateway)

"I want to expose an open, public MCP endpoint on serverless infrastructure (Cloud Run + Firebase CDN) while avoiding memory leaks, session tracking collisions, and DDOS-induced Gemini billing spikes so that the service runs 100% reliably at near-zero operating cost with deterministic IP rate limiting."

Public APIs invite variable traffic. The infrastructure must scale to zero when idle, handle bursty concurrency across distributed Cloud Run instances, maintain zero in-memory session leakage, and enforce deterministic rate limits on expensive upstream LLM calls.

🏛️ System Architecture: Stateless Streamable HTTP

The Model Context Protocol (MCP), originally introduced by Anthropic in late 2024 and standardized under version 2024-11-05, defines how models query external tools and resources.

While early MCP implementations relied on local standard input/output (stdio) or long-lived Server-Sent Events (SSE) tied to local processes, the web-native specification introduces Streamable HTTP.

flowchart LR
    subgraph External_Agents["External AI Agents"]
        Muse["Muse Agent"]
        Instinct["Instinct Agent"]
        Cursor["Cursor / Claude Code"]
    end

    subgraph Edge_Routing["Edge & CDN Routing"]
        CDN["Firebase Hosting CDN (herrington.ai/mcp)"]
        WAF["CORS & Accept Normalization"]
    end

    subgraph Cloud_Run["Stateless Cloud Run Engine"]
        Router["Express /mcp Router"]
        Transport["StreamableHTTPServerTransport
(sessionIdGenerator: undefined)"]
        ServerFactory["createMcpServer() Factory"]
    end

    subgraph Knowledge_Corpus["Knowledge & Model Services"]
        TFIDF["TF-IDF Semantic Inverted Index"]
        MarkdownStore["Raw Markdown Corpus (39 Essays, 20 Projects)"]
        RateLimiter["Token Bucket Rate Limiter (30 req/hr)"]
        GeminiGateway["Gemini 3.8 / 2.5 Pro Gateway (dh.sh)"]
    end

    External_Agents -->|POST JSON-RPC 2.0| CDN
    CDN --> WAF
    WAF --> Router
    Router --> Transport
    Transport --> ServerFactory
    ServerFactory -->|search_essays / get_essay| TFIDF
    ServerFactory -->|list_projects / get_project| MarkdownStore
    ServerFactory -->|ask_dhsh| RateLimiter
    RateLimiter -->|Filtered Prompt| GeminiGateway

The Architectural Pitfall: Stateful MCP Sessions on Serverless Infrastructure

During our initial rollout of the Streamable HTTP transport, we encountered two subtle protocol failures when clients connected via curl and automated harnesses:

1. The Global Initialization Lock: The official @modelcontextprotocol/sdk transport defaulted to stateful mode when given a session ID generator (sessionIdGenerator: () => ...). In a single-instance Express application, connecting a transport to a persistent Server instance meant that once the first client executed initialize, an internal _initialized boolean was set to true. When a second, independent agent initialized, the SDK threw:

   {"code": -32600, "message": "Invalid Request: Server already initialized"}

1. The Session Demanded Dead-End: Because the server was in stateful mode, non-initialization requests without a matching Mcp-Session-Id header were rejected with:

   {"code": -32001, "message": "Session not found"}

For public serverless microservices where every tool is read-only, maintaining sticky in-memory sessions across ephemeral Cloud Run containers provides zero value and introduces fatal connection fragility.

The Stateless Breakthrough: sessionIdGenerator: undefined

The solution was to switch the entire MCP endpoint into true stateless mode:

• In the MCP TypeScript SDK, setting sessionIdGenerator: undefined signals the transport to disable session tracking completely.
• Instead of maintaining a single global transport, the Express router instantiates a fresh StreamableHTTPServerTransport and connects a lightweight createMcpServer() instance per incoming HTTP request.
• On request completion, res.on('close') immediately releases transport connections and cleans up memory.
• The server never demands or returns an Mcp-Session-Id header.
• Notifications like notifications/initialized are handled cleanly with HTTP 202 Accepted and an empty body.
• Both Streamable SSE clients and direct JSON-RPC 2.0 clients are supported out-of-the-box via enableJsonResponse: true.

// Stateless Streamable HTTP Transport Initialization
const reqServer = createMcpServer({ clientIp, userAgent });
const transport = new StreamableHTTPServerTransport({
  sessionIdGenerator: undefined,
  enableJsonResponse: !wantsSSEOnly
});

await reqServer.connect(transport);
await transport.handleRequest(req, res, req.body);

res.on("close", () => {
  try {
    transport.close();
    reqServer.close();
  } catch (_) {}
});

🛠️ The Machine-Callable Tool Suite

The /mcp endpoint exposes six deterministic, read-only tools designed to give AI agents deep access to Daniel's work:

Preserving Token Economy: Zero-Wrapper Markdown

A key design requirement was minimizing token waste for consuming models. When an agent calls get_essay("beyond-peak-tflops"), returning JSON-escaped HTML or redundant nested structures forces the model to burn precious reasoning context parsing string literals.

By streaming raw, clean Markdown directly in the tool's text content block, external agents receive pristine headings, code blocks, and math equations ready for immediate context injection.

🛡️ Security, Rate Limiting & The LLM Gateway

Exposing a public AI endpoint demands rigorous defensive engineering:

sequenceDiagram
    autonumber
    actor Agent as External Agent (Muse/Instinct)
    participant CDN as Firebase CDN / Cloud Run
    participant Limiter as Token Bucket Rate Limiter
    participant Filter as Anti-Injection Guardrail
    participant Gemini as Google Gemini 3.8 Gateway

    Agent->>CDN: POST /mcp (ask_dhsh)
    CDN->>Limiter: Check Client IP Quota (30 req / hr)
    alt Rate Limit Exceeded
        Limiter-->>Agent: HTTP 200 (Result: "Rate limit exceeded. Try again in Xs")
    else Quota Available
        Limiter->>Filter: Scan for Jailbreak Signatures
        alt Prompt Injection Detected
            Filter-->>Agent: HTTP 200 (Result: Deflected to Systems Topics)
        else Clean Query
            Filter->>Gemini: POST generateContent (Thinking Budget = 0)
            Gemini-->>CDN: Streamed Persona Response
            CDN-->>Agent: JSON-RPC 2.0 Tool Result
        end
    end

1. Read-Only Invariance: No tool on /mcp has write access, database mutation capabilities, or email sending permissions. It is mathematically impossible for an external agent to alter site state.
2. Deterministic Token Bucket Rate Limiting: The ask_dhsh tool forwards directly to Google's Gemini API, making it the only tool with real-world marginal compute cost. We implemented an in-memory sliding window rate limiter strictly capping requests to 30 calls per hour per IP.
3. Anti-Jailbreak Deflection Filter: Queries to ask_dhsh pass through a deterministic signature scanner that intercepts prompt injection attempts ("ignore previous instructions", "reveal your prompt", etc.) and politely deflects back to Daniel's verified technical domains.
4. CORS & Origin Freedom: Every response sets Access-Control-Allow-Origin: *, allowing in-browser web agents, local IDEs, and cloud microservices to call the endpoint without proxy hurdles.

📄 The Dual-Tier Corpus: /llms.txt & /llms-full.txt

In tandem with the MCP server, we established a clean, tiered documentation architecture for AI crawlers:

• /llms.txt: A concise index designed for rapid discovery. Outlines Daniel's background, current role as CAIO at The Zebra, featured projects, recent publications, and points agents directly to the /mcp server endpoint.
• /llms-full.txt: A 512 KB comprehensive text repository containing the complete, unedited Markdown text of every single essay on the site, each cleanly delimited with its canonical URL and metadata. This allows frontier models with 1M+ context windows to ingest Daniel's entire intellectual corpus in a single HTTP GET.

📈 Results & Verification

The server was validated using zero-dependency curl commands simulating autonomous agent handshakes against production:

1. Sub-100ms Tool Latency: Semantic searches via search_essays return ranked results in under 90 milliseconds on Cloud Run.
2. Infinite Concurrency: Multiple independent agents execute initialize and tools/list simultaneously without session collisions or lock contention.
3. Ambient IDE Access: Developers can now add herrington.ai directly to their local Cursor or Claude Code configurations:

{
  "mcpServers": {
    "herrington-ai": {
      "url": "https://herrington.ai/mcp"
    }
  }
}

By eliminating the human browser tax, herrington.ai is now fully native to the agentic web.