ANAN EVALUATION · EVIDENCE REPORT

Historical MCP cost comparison

5 real App runs per version. Answers were not independently graded; this is not a single-factor BM25 ablation.

Evidence snapshot · 2026-10-17

GitHub MCP: tool-discovery cost comparison

The same task, “我的github有多少star” · DeepSeek deepseek-flash · 5 real App runs per version.

The old MCP interface exposed list / search / describe / call inside a tool. The newer version uses unified tool_search / tool_describe / tool_call with BM25. The old version also supported on-demand schema discovery and reuse; it did not expose every schema upfront.

Mean total input + output · tokens

102,447

Old MCP951bd96 · n=5

90,406

Unified discovery + BM25888946d · n=5

Mean model requests · count

8.0

Old MCP951bd96 · n=5

8.6

Unified discovery + BM25888946d · n=5

Mean task duration · seconds

42.47

Old MCP951bd96 · n=5

48.65

Unified discovery + BM25888946d · n=5

This section is a cost observation: means include all historical samples accepted for measurement with complete usage, regardless of answer correctness. Missing usage is not zero. This differs from the successful-run means above. Total input already includes cached input; Session totals are not added again. Duration excludes build and environment setup. Each chart has its own zero-based scale; bar heights cannot be compared across charts.

Observed tradeoffs:The newer version used fewer mean total tokens, but made more model requests and took longer. None of the 10 answers was independently graded: completion or process compliance does not establish that the star count was correct. The versions differ in other ways and were not run in interleaved order. GitHub data, the hosted catalog and provider caches were not frozen. These observations cannot be attributed solely to BM25 or search / describe / call, and do not establish statistical significance or monetary savings.

Old MCP: expand all historical samples

commit 951bd96b3bb1ca4a0f6691261febe480121d9fc1

Run	Execution	Process	Answer	Total tokens	Model requests	s	list / search / describe / call	Usage	Cleanup / App reset
1	completed	satisfied	unscored	77,244	7	35.28	0 / 1 / 2 / 2	complete	clean / clean
2	completed	satisfied	unscored	144,535	10	53.61	2 / 0 / 2 / 3	complete	clean / clean
3	completed	satisfied	unscored	105,225	8	43.46	1 / 3 / 2 / 3	complete	clean / clean
4	completed	satisfied	unscored	101,534	8	43.07	2 / 0 / 2 / 3	complete	clean / clean
5	completed	satisfied	unscored	83,699	7	36.95	1 / 0 / 2 / 3	complete	clean / clean

Historical aggregate source SHA256:eab1ae8589fe77c13d61acc65c969c8d10934a78245dea251320382c791d0c10

Unified discovery + BM25: expand all historical samples

commit 888946dab2836f9263d4d6037051e6903ad2ad11

Run	Execution	Process	Answer	Total tokens	Model requests	s	list / search / describe / call	Usage	Cleanup / App reset
1	completed	satisfied	unscored	121,543	11	62.59	0 / 6 / 2 / 3	complete	clean / clean
2	completed	satisfied	unscored	85,199	8	43.52	0 / 3 / 2 / 3	complete	clean / clean
3	completed	satisfied	unscored	93,506	9	44.10	0 / 2 / 2 / 3	complete	clean / clean
4	completed	satisfied	unscored	68,775	7	50.42	0 / 3 / 2 / 2	complete	clean / clean
5	completed	satisfied	unscored	83,009	8	42.60	0 / 2 / 2 / 3	complete	clean / clean

Historical aggregate source SHA256:d8dc48b5d1c74e3d05682a1a9f9a309495c62d5e7cbdad7b12f67c7df6e72c77

These 10 additional historical cost observations are shown separately. They are not included in the 78 selected task records above or the successful-context means. Account details, repository contents, raw Sessions and credentials are neither read nor published here. No reruns were performed for this report.

Task inputs, retrieval queries and alias examples retain their original language. Downloads preserve the original evaluation data.