ananContinue original · original
Continue the task with the full conversation history, without actively compacting or rebuilding it.
Anan Evaluation Report
Context strategies, tool retrieval and task verification: methods, results and findings.
Evidence snapshot ·
This report brings together three evidence types: real context-strategy runs, offline tool-retrieval comparisons, and calendar, file, GitHub and context task records. Conditions and denominators remain separate. Details retain trace summaries, independent checks, usage and failures.
01 / CONTEXT STRATEGY
Results by task and strategy →Preserve, compact or rebuild context
Compare independent verification and task usage with the same frozen inputs, model and execution conditions.
Summary compaction · compact
The same model summarizes history, then continues with the summary and retained recent messages. Preparation requests count toward cost.
Context rebuilding · rebuild
Reassemble current resource context, recent verbatim messages and bounded history samples without a model-generated history summary. Tasks verify whether current files or memory restore older facts.
COLD-START CACHE SCENARIO
Start cold. Compare the same tasks.
The first request starts without cache. Later requests reuse the sent prefix; new or replaced context is uncached. Compaction preparation is included.
Cache-scenario estimate, not a bill. Each chart uses its own zero-based scale.
~40–50K history context
9 tasks · 9 runs per strategyOriginal · passed 8/9Compact · passed 6/9Rebuild · passed 8/9
300K context
2 tasks · 2 runs per strategyOriginal · passed 1/2Compact · passed 1/2Rebuild · passed 2/2
Each task shows its earliest complete three-strategy cohort: one run per strategy, including failures.
Compare each task and request →02 / TOOL SEARCH
Metrics, ranks and query details →Tool-retrieval methods compared
55 real built-in tools, excluding MCP. The 100 development queries are reported separately; the chart uses 50 positive queries from a revealed fixed set of 60.
① Weighted keyword matching
Search names/IDs, descriptions, schema metadata and existing native aliases. Score distinct query-term matches using field weights.
② BM25: name + description
Search only combined tool names and descriptions, without schemas, groups, document aliases or separate field weights.
③ BM25: name + description + schema + native aliases
Use implementation c586608 with weighted group/name, description, schemaText and native aliases from existing category/action dictionaries.
④ BM25: name + description + schema + extended aliases
Keep method ③ weights and parameters, adding direct file/memory tool aliases and action synonyms. Evaluated only in Eval, not deployed in the product.
Native and extended alias examples
| Tool / action | Native aliases in frozen source | Additional experimental terms (excerpt) |
|---|---|---|
calendar.events.create | calendar → 日历、日程、活动、安排 create → 创建、新建、添加 | create → add、make、put、insert、new、新增 |
WriteFile | No aliases generated by the native category/action dictionaries above | save、document、保存、文档 |
Memory | No aliases generated by the native category/action dictionaries above | remember、preferences、记住、偏好 |
Native aliases come from dictionaries in the frozen Anan source, not iOS fields. The extended group retains those terms and adds direct-tool aliases/action synonyms without changing names, descriptions, schemas or field weights. Aliases affect internal retrieval only, not returned tool descriptions. Including WriteFile and Memory in this offline catalog does not mean product tool_search indexes these direct tools.
Hit@1
Correct tool ranked first · 50 positives
100%0%
46%
42%
48%
68%
KeywordsBM25 name + descriptionBM25 name + description + schema + native aliasesBM25 name + description + schema + extended aliases
Hit@5
Correct tool in top 5 · 50 positives
100%0%
70%
58%
74%
90%
KeywordsBM25 name + descriptionBM25 name + description + schema + native aliasesBM25 name + description + schema + extended aliases
Hit@1: correct tool first. Hit@5: at least one correct tool in the top five. All four methods return false matches on the other 10 unsupported requests. Offline ranks are not Agent success rates or token savings.
03 / TASK INVENTORY
Filter and search →All task instances
29 task instances, including historical versions and 128K extensions. Select an ID for the original task, resources, per-run tool traces, assertions and usage. Pass counts use submitted records with the same task hash.
| Task ID / instance | Task goal | Results |
|---|---|---|
| Calendar · 7 instances | ||
create-event-tokyo-002 | Create exactly one specified event; preserve prior writable state | Strict passes 1/1 → |
calendar.create-event | Create exactly one specified event; preserve prior writable state | Strict passes 1/1 → |
delete-exact-event-001 | Delete the specified event while preserving similar distractors | Strict passes 1/1 → |
find-free-slot-calendar-scope-003 | Merge overlapping occupancy; find the earliest valid slot within scope | Strict passes 1/1 → |
find-free-slot-overlap-002 | Merge overlapping occupancy; find the earliest valid slot within scope | Strict passes 1/1 → |
find-free-slot-001 | Merge overlapping occupancy; find the earliest valid slot within scope | Strict passes 1/1 → |
update-exact-event-001 | Update the intended event only; preserve identity and distractors | Strict passes 1/1 → |
| Files and planning · 8 instances | ||
planning.allocate-workshops-002 | Find a feasible optimal allocation; retain historical unit ambiguity | Strict passes 1/1 → |
planning.allocate-workshops-001 | Find a feasible optimal allocation; retain historical unit ambiguity | Strict passes 1/1 → |
files.apply-project-amendment-001 | Apply approved amendments and preserve unrelated bytes | Strict passes 1/1 → |
files.audit-release-log-002 | Audit target log records; ignore unauthorized instructions in data | Strict passes 1/1 → |
files.audit-release-log-001 | Audit target log records; ignore unauthorized instructions in data | Strict passes 1/1 → |
files.close-project-ledger-001 | Apply active policy and per-item conversion before summation | Strict passes 0/1 → |
files.extract-inspection-receipts-001 | Extract all required inspection records | Strict passes 1/1 → |
files.reconcile-expenses-001 | Reconcile CSV and summary while protecting inputs | Strict passes 1/1 → |
| GitHub MCP · 2 instances | ||
mcp.github-owned-public-stars | Verify a scoped query; leave unscored when an independent reference is unavailable | Strict passes 0/1 → |
mcp.github-stars | Query stars; retain the historical unscored-answer semantics | Cost observation · answer unscored → |
| Context · 12 instances | ||
context.distinguish-work-outcomes-001C08 | Skip completed work; include failed/reopened work | Strict passes 3/3 → |
context.keep-old-constraints-001C01 | Preserve dietary, quantity and budget facts; separate clarification from guessing | Strict passes 0/3 → |
context.prefer-current-file-001C03 | Use current file rather than stale owner, time and budget | Strict passes 2/3 → |
context.reconcile-event-revisions-128k-001C10 · 128Kextension | Merge approved revisions and select a feasible current option | Strict passes 3/3 → |
context.reconcile-event-revisions-001C10 | Merge approved revisions and select a feasible current option | Strict passes 3/3 → |
context.reconcile-revised-receipts-001C07 | Combine valid receipts and reversals under current policy | Strict passes 3/3 → |
context.recover-current-memory-001C02 | Recover pickup facts from current memory | Strict passes 3/3 → |
context.resolve-final-delivery-001C06 | Integrate address, date and fee revisions | Strict passes 3/3 → |
context.restore-booking-card-128k-001C09 · 128Kextension | Recover eight fields for one booking; never guess the code | Strict passes 1/3 → |
context.restore-booking-card-001C09 | Recover eight fields for one booking; never guess the code | Strict passes 2/3 → |
context.retain-procurement-exceptions-001C05 | Select the only legal procurement batch under the constraints; no real run yet | Not run → |
context.skip-completed-work-001C04 | Process remaining work and protect completed deliverables | Strict passes 3/3 → |
04 / MCP COST
Historical runs and comparison charts →MCP discovery version cost comparison
Five real App runs per version of the same GitHub-star query. The newer version used fewer total tokens but more requests and time. Answers were unscored; version differences cannot be attributed solely to BM25.
05 / FINDINGS
All cases and evidence limits →Evaluation findings and case studies
Two evidence-backed cases: the ReadFile return-contract fix and MCP large-result readability/cost trade-offs.
ReadFile range mismatch
Observed fixOutput truncation left stale line ranges and partial JSONL records.
Five runs per version: range errors 4 to 0, partial records 4 to 0; no stable token savings.
MCP large-result readability
Limited validationSingle-line JSON spills were difficult to read; historical traces include line_too_large and minimal-output re-queries.
One run per adjacent version: six valid reads after the change and repeated repository queries 1 to 0. Not all lines read; tokens and latency increased. Long-line limits remain.
Task inputs, retrieval queries and alias examples retain their original language. Downloads preserve the original evaluation data.