anan.Tasks, environments, verifiers and evidence are recorded separately. Completion does not mean correctness; missing usage is not zero.
Evidence snapshot ·
Beyond resource families, tasks are classified by purpose: regression protects existing behavior, capability/stress probes limits, and cost observation measures usage/process only.
| Purpose | Capability and task | User goal | Independent verification and evidence |
|---|---|---|---|
| Regression | State creation and time zonescreate-event-tokyo-002 calendar.create-event | Create exactly one specified event; preserve prior writable state | Same-simulator EventKit snapshots; count, fields, target calendarrunner/calendar.py:verify_createVerifier self-check: tests/test_runner.py:test_wrong_fields_and_extra_event |
| Regression | Exact-target discrimination and updateupdate-exact-event-001 | Update the intended event only; preserve identity and distractors | Baseline fixture binding and final EventKit staterunner/calendar.py:verify_extendedVerifier self-check: tests/test_runner.py:test_baseline_mutation_and_removal |
| Regression | Exact deletion and preservationdelete-exact-event-001 | Delete the specified event while preserving similar distractors | Target absence and preservation of other captured events/fieldsrunner/calendar.py:verify_extendedVerifier self-check: tests/test_runner.py:test_baseline_mutation_and_removal |
| Regression | Interval planning and resource scopefind-free-slot-calendar-scope-003 find-free-slot-overlap-002 find-free-slot-001 | Merge overlapping occupancy; find the earliest valid slot within scope | Independent host interval oracle; no prescribed tool routerunner/calendar.py:earliest_slotVerifier self-check: tests/test_runner.py:test_window_mismatch_is_not_graded |
| Regression | Deduplication and monetary reconciliationfiles.reconcile-expenses-001 | Reconcile CSV and summary while protecting inputs | Independent calculation from receipts/charges and actual output parsingrunner/files.py:verifyVerifier self-check: tests/test_files.py:test_wrong_amount_duplicate_id_invoice_and_source_changes_fail |
| Capability / stress | Multi-table joins, refunds and FXfiles.close-project-ledger-001 | Apply active policy and per-item conversion before summation | Independent business reference; validate amounts and record setsrunner/ledger.py:referenceVerifier self-check: tests/test_challenges.py |
| Capability / stress | Joint constrained optimizationplanning.allocate-workshops-002 planning.allocate-workshops-001 | Find a feasible optimal allocation; retain historical unit ambiguity | Independent enumeration of feasible optima and output constraintsrunner/planning.py:referenceVerifier self-check: tests/test_challenges.py |
| Capability / stress | Instruction boundaries and precise mutationfiles.apply-project-amendment-001 | Apply approved amendments and preserve unrelated bytes | Reference-approved values and explicitly bounded mutation pathsrunner/amendment.py:referenceVerifier self-check: tests/test_diagnostics.py |
| Capability / stress | Exhaustive retrieval and distractor resistancefiles.audit-release-log-002 files.audit-release-log-001 | Audit target log records; ignore unauthorized instructions in data | Independent full-log filtering, without rewarding fixed read countsrunner/release_log.py:referenceVerifier self-check: tests/test_diagnostics.py |
| Capability / stress | Paged reading and long-record fidelityfiles.extract-inspection-receipts-001 | Extract all required inspection records | Input reference versus actual outputs; tool contract checked separatelyrunner/inspection.py:referenceVerifier self-check: tests/test_inspection.py |
| Capability / stress | Old constraints / information-loss riskcontext.keep-old-constraints-001 | Preserve dietary, quantity and budget facts; separate clarification from guessing | Host oracle audited against frozen history/current resources; actual CSV and protected state, not self-reportrunner/context_cases.py:verifyVerifier self-check: tests/test_context_strategy.py:test_all_tasks_validate_correct_empty_wrong_and_protected_inputs |
| Capability / stress | Recover facts from memorycontext.recover-current-memory-001 | Recover pickup facts from current memory | Host oracle audited against frozen history/current resources; actual CSV and protected state, not self-reportrunner/context_cases.py:verifyVerifier self-check: tests/test_context_strategy.py:test_all_tasks_validate_correct_empty_wrong_and_protected_inputs |
| Capability / stress | Current resource supersedes stale historycontext.prefer-current-file-001 | Use current file rather than stale owner, time and budget | Host oracle audited against frozen history/current resources; actual CSV and protected state, not self-reportrunner/context_cases.py:verifyVerifier self-check: tests/test_context_strategy.py:test_all_tasks_validate_correct_empty_wrong_and_protected_inputs |
| Capability / stress | Avoid repeating completed workcontext.skip-completed-work-001 | Process remaining work and protect completed deliverables | Host oracle audited against frozen history/current resources; actual CSV and protected state, not self-reportrunner/context_cases.py:verifyVerifier self-check: tests/test_context_strategy.py:test_all_tasks_validate_correct_empty_wrong_and_protected_inputs |
| Capability / stress | Retention of exceptionscontext.retain-procurement-exceptions-001 | Select the only legal procurement batch under the constraints; no real run yet | Host oracle audited against frozen history/current resources; actual CSV and protected state, not self-reportrunner/context_cases.py:verifyVerifier self-check: tests/test_context_fidelity.py:test_every_output_field_matters |
| Capability / stress | Combine revisions and exceptionscontext.resolve-final-delivery-001 | Integrate address, date and fee revisions | Host oracle audited against frozen history/current resources; actual CSV and protected state, not self-reportrunner/context_cases.py:verifyVerifier self-check: tests/test_context_fidelity.py:test_every_output_field_matters |
| Capability / stress | Recompute revised totalscontext.reconcile-revised-receipts-001 | Combine valid receipts and reversals under current policy | Host oracle audited against frozen history/current resources; actual CSV and protected state, not self-reportrunner/context_cases.py:verifyVerifier self-check: tests/test_context_fidelity.py:test_every_output_field_matters |
| Capability / stress | Distinguish done, failed and reopened workcontext.distinguish-work-outcomes-001 | Skip completed work; include failed/reopened work | Host oracle audited against frozen history/current resources; actual CSV and protected state, not self-reportrunner/context_cases.py:verifyVerifier self-check: tests/test_context_fidelity.py:test_every_output_field_matters |
| Capability / stress | Exact fact retention in dense historycontext.restore-booking-card-128k-001 context.restore-booking-card-001 | Recover eight fields for one booking; never guess the code | Host oracle audited against frozen history/current resources; actual CSV and protected state, not self-reportrunner/context_cases.py:verifyVerifier self-check: tests/test_context_dense.py:test_correct_and_each_field_corruption |
| Capability / stress | Cross-turn revisions and constrained planningcontext.reconcile-event-revisions-128k-001 context.reconcile-event-revisions-001 | Merge approved revisions and select a feasible current option | Host oracle audited against frozen history/current resources; actual CSV and protected state, not self-reportrunner/context_cases.py:verifyVerifier self-check: tests/test_context_revisions.py:test_task_correct_result_and_each_field_corruption |
| Cost observation | Tool discovery and task costmcp.github-stars | Query stars; retain the historical unscored-answer semantics | Required-operation checks and ledger cost do not prove answer correctnessrunner/observation.py:verifyVerifier self-check: tests/test_mcp_checkpoint.py:test_new_task_preserves_old_semantics |
| Regression | Scoped external-answer verificationmcp.github-owned-public-stars | Verify a scoped query; leave unscored when an independent reference is unavailable | Independent GitHub API, complete pagination and temporal checks; saved TLS failure is not a verified answer passrunner/observation.py:verifyVerifier self-check: tests/test_mcp_checkpoint.py:test_answer_matches_wrong_changed_incomplete_time_and_ambiguity |
Purpose is not measured difficulty. A task definition or offline test is not a real pass. C05 has no real run; the GitHub-answer task still has an independent-oracle gap. Select task IDs for actual results.
Corrupt individual fields to test critical facts; use empty files, duplicate rows and wrong columns to check format/sets; recompute the oracle after resource changes; protect sources and nontargets. These are offline negative cases, not additional Agent runs.
A correct final file does not prove the process avoided unauthorized writes. C04/C08 check only enabled tools and specified protected paths; incomplete traces stay unknown. Calendar checks cover only captured fields in 2029–2031.
Completion, correctness, observability and cleanup each have separate acceptance.
Calendar checks read EventKit state on the same device. File checks compare actual outputs and source protection. Context tasks also check configuration, inputs and protected operations. GitHub cost observations remain unscored; answer tasks require independent API references. Missing/changed references or unextractable answers stay unscored.
Context/MCP primarily use the request ledger, including compaction preparation. Older calendar/file samples use saved Session responses. They are not summed twice. Cached tokens are part of input and reasoning is not added to output again. Failures, cancellations and unsaved requests can leave gaps.
No second Agent, physical-device or overall non-inferiority claim. Calendar coverage is limited to captured fields in 2029–2031. Tool steps summarize original traces; hashes identify private originals. This package cannot independently replay the environment or reproduce business oracles.
This page includes the recent non-context suite and all selected context records, not every earlier MCP/ReadFile run. The report does not connect to evaluation services or execute trace content. The website’s existing visitor analytics are described in the privacy policy.
Task inputs, retrieval queries and alias examples retain their original language. Downloads preserve the original evaluation data.