2026-09-22 · benchmark

MCP benchmark: seven models, one format

Versione italiana

Controlled comparison of an MCP server implementing the Mosaix Format against a monolithic file baseline. Seven LLMs, four corpus sizes, nine test dimensions. The architectural advantage holds across models — with two instructive exceptions.

Setup

The benchmark compares two arms on the same task. The MCP arm uses a server implementing the Mosaix Format v1.2.0 specification — 11 tools including search, read_note, write_note, compose, and check. Notes are atomic: one idea per note, self-describing frontmatter (summary, keywords, entities, links), BM25 search. The standard arm uses a single file containing the same content as a monolithic document, with read_file and search_file as the only tools.

The corpus is an industrial product catalog (hydraulic hose, fittings, couplings) at four sizes: 35 notes, 65 notes (258 KB), 143 notes, and 558 notes (1 MB). The monolith for the 558-note vault is a 748 KB file.

Seven LLMs were tested via OpenRouter: DeepSeek V4 Flash ($0.06/$0.14 per million tokens), GPT-5.6 Luna ($0.20/$1.20), Ministral 8B ($0.15/$0.15), GLM 5.3 Flash ($0.075/$0.25), Gemini 3.8 Flash ($0.75/$3.75), Qwen 3.8 Flash ($0.15/$0.47), and DeepSeek V4.1 Flash ($0.10/$0.40). Partial data exists for Claude Opus 4.8 and GPT-5.6 Sol (credits exhausted mid-run).

Every run uses a cold start: a nonce is injected into the system prompt to defeat prefix caching. All token counts are uncached. Each test is N=1 per arm per model.

Answer quality

Three independent LLM judges rated the same six answers from both arms on a 1–5 scale for accuracy, completeness, and relevance. The judges had no knowledge of which arm produced which answer.

Full concordance across three judges from different providers. The quality gap (+1.5 points) is not marginal — it is the difference between an incomplete answer that omits key specifications and a comprehensive answer that cites specific product codes, pressure ratings, and temperature ranges. The monolith arm suffers because the model receives the entire document and must locate the relevant section; the MCP arm retrieves only the relevant notes.

RAG multi-turn cost

Six consecutive questions over the catalog, cumulative context. The model must search, read, and answer — each turn builds on the previous. This is the core cost benchmark: it measures real-world retrieval cost over a sustained session.

65 notes (full catalog)

MCP costs 6–56% of the monolith on five of six models. Gemini 3.8 Flash is the exception: it generates fewer tool calls but consumes more tokens per MCP response, making the atomic arm 67% more expensive. The finding is model-specific, not architectural — Gemini’s high per-token pricing ($0.75/$3.75) amplifies even small token differences.

MCP tool calls stay in a narrow band (20–44) regardless of model. Monolith tool calls range from 44 to 600, depending on how aggressively the model retries search_file on a large document. The consistency of the MCP arm suggests the advantage comes from the architecture, not from model-specific behavior.

558 notes (8.5× larger)

DeepSeek fails all six questions on the monolith (max_loops on every query). MCP answers all six for $0.05. Luna completes everything on both arms but MCP costs 40% less.

No-reasoning mode

Same 65-note RAG task with chain-of-thought reasoning disabled. Six models tested. Disabling reasoning amplifies the MCP advantage because it shrinks output tokens (where MCP has no edge) while leaving input tokens (where MCP saves the most) unchanged.

Luna no-reasoning at 0.03× is the best ratio recorded: MCP costs 97% less than the monolith. The median across six models is 0.28× (vs 0.38× with reasoning enabled). Every model favors MCP in no-reasoning mode — including Gemini, which inverted the ratio with reasoning on.

How costs scale with corpus size

DeepSeek V4 Flash on the same six questions at three corpus sizes:

MCP cost grows from $0.043 to $0.050 as the corpus scales 8.5× — a 16% increase. The monolith cost stays roughly flat in dollars, but the completion rate drops from 4/6 to 0/6. At 558 notes, the monolith arm cannot answer a single question within the loop limit. Cost stability without completion is not a useful metric.

Cross-model summary

Green cells are where MCP costs less; red where it costs more. The pattern: MCP wins on 27 of 29 measured cells. The two losses have specific, documented causes (small-model loop explosion at scale; high-price model on short responses). The architecture works. The exceptions tell you where to look.

Stress test: out-of-domain queries

Six deliberately difficult questions on the 558-note vault: a product that does not exist, a domain the catalog does not cover, an ambiguous question with no context, an impossible comparison, an incorrect product code, and a physically impossible specification. These questions are designed to test how quickly the system recognises that the answer is not in the data.

The monolith arm cannot distinguish “I haven’t found it yet” from “it is not here”. On a 748 KB file, the model keeps calling search_file with different patterns, hoping the next search will find the answer. DeepSeek times out on four of six questions. The MCP arm gets a definitive signal from the BM25 search: no results means the data is not in the vault. The model accepts this and responds in 1–4 tool calls.

Other dimensions

Three additional benchmarks complete the picture: latency, ingestion cost, and CRUD lifecycle operations.

Latency

End-to-end time for the six-question RAG session:

Faster on every model and corpus size. One exception: Ministral 8B on 558 notes takes 4× longer with MCP, consistent with the excessive search loops documented above. The speedup comes from reading less: fewer tokens in means less time waiting for the model to process them.

Ingestion

The one-time cost of converting the raw catalog into atomic Mosaix notes, amortised over all subsequent queries:

Luna produces nearly twice as many notes as DeepSeek (60 vs 32) at a quarter of the cost ($0.33 vs $1.44) in a tenth of the time. A model with a higher per-token price can be dramatically cheaper in total when it plans better and finishes in fewer rounds.

CRUD

Create, read, search, update, supersede (following the Mosaix R7 rule: notes are superseded, not deleted), and verify conformity. Near parity across the board: 0.92×–1.15× on warm runs. The cold-start advantage (0.42×) disappears with prefix caching, which tells you exactly where the MCP advantage lives: in the input tokens, not the operations.

Caveats

These results should be read with the following limitations in mind.

N=1 per arm per model per test. The measurements are directional, not statistically powered. A single outlier run could shift any individual ratio.

All models run through OpenRouter, which does not expose prefix cache hit rates. Every token is counted as uncached. In production, prefix caching would reduce the monolith’s cost — but also some of MCP’s advantage, since the monolith benefits more from caching (it re-reads the same large document repeatedly).

The corpus is an industrial product catalog. Results may differ on creative, conversational, or code-generation tasks. The previous study (Why one note at a time) covered code generation; this one covers knowledge retrieval.

The 558-note vault contains approximately 133 real notes and 415 synthetic notes generated to stress the search index. The synthetic notes are structurally valid but may not represent real-world note density and overlap patterns.

The benchmark author designed both the format and the test. We report both the wins and the losses (Gemini and Ministral at scale) and encourage independent replication.

Further reading

Why one note at a time — the earlier code-generation study, measuring token compression and local model performance.

Mosaix Format v1.2.0 specification — the full spec, including the 11 MCP tools benchmarked here.

GitHub repository — specification, reference checker, and example vaults.