ANAN EVALUATION · EVIDENCE REPORT

calendar.create-event

Task inputs, individual runs, tool traces, independent verification and usage.

Evidence snapshot ·

calendar.create-eventcalendar · v0.3Strict passes 1/1 submitted · not submitted 0Passed1 records＋

Capability: State creation and time zones
Goal: Create exactly one specified event; preserve prior writable state
Independent verification: Same-simulator EventKit snapshots; count, fields, target calendar

User task

Create a calendar event titled Benchmark Planning in the Agent Eval calendar on January 15, 2030, from 15:00 to 15:30 UTC. Do not modify any other events.

Resources and protocol

ios.eventkit.calendar · state-verification

Default task timeout: 180s; individual samples may have approved overrides. Counts include only matching task hashes. The badge shows the latest submitted state; historical versions remain separate. Repeats use different frozen clocks/conditions and do not establish a stable pass rate.

  • Resources come from independent fixtures or frozen context; the full environment is not included in this package.
sample-c954db77488bec0dPassed27,805 tokens
Version 0.3 · commit ec6bb29448b2 · timeout 180s · yoloSubmitted · execution Completed · 46.87 s
Independent goal ✓ PassedConfiguration Unknown · ledger attribution Unknown

Tool trace saved

Only the recorded tool-call order is shown, without arguments, result bodies or reasoning. This is not a full reasoning trace.

  1. 01tool_searchsuccess
  2. 02tool_describesuccess
  3. 03tool_describesuccess
  4. 04tool_callsuccess
  5. 05tool_callsuccess

Independent verification 10 assertions

  • ✓ Passedbaseline_owned_calendar_empty
  • ✓ Passedfixture_calendar_unchanged
  • ✓ Passedexactly_one_new_event
  • ✓ Passedbaseline_events_unchanged
  • ✓ Passedcalendar_metadata_unchanged
  • ✓ Passedevent_title
  • ✓ Passedevent_start
  • ✓ Passedevent_end
  • ✓ Passedevent_isAllDay
  • ✓ Passedevent_calendar

Usage Saved Session responses (some requests may be missing)

Complete Coverage: known records / saved records; not a complete billing statement.

MetrictokensCoverage
Total input27,0365/5
Of which cached input25,3445/5
Uncached input1,6925/5
Output7695/5
Input + output27,8055/5

Model requests — · saved responses 5
Preparation / continuation requests —／—. Cached input is included in total input; reasoning is not added to output again.

Resource cleanup Done App reset Done Simulator reset Unknown Shutdown Done
Source identity and evidence hashes
Run started
2026-10-12T12:46:39.139837+00:00
task
58a6df4bbc63333cc791b73f8a6b38a0e9bac542877450386a265ce45cab1a07
binary
2296cead77ac77274cff68e9234f903a3af4cee768d2b97d8b42f16928dc72c9
Eval execution
None
cohort
None
model
{'provider': 'DeepSeek', 'model': 'deepseek-flash', 'thinking': 'not_recorded'}
manifest
979e58626f0fd40cbeb0d956fc2a3cf8f57f3bb49f9963ee1b926f47c4073886
result
f9ede5f100553dfd31c0b64b6ac9883fb5e505cae960c57bf65d9c5f808d7339
run
b4791f9261f42103df787ccd380bf1b22abf22227aca70e1f97fa93c32224ae2
evidence
470aba3af559918d6aedcaa343cd971516b0bfc426ef350fb5b3e6437357cea6
usage
35dfa349fb8516db0998bcfdd595b68dec27e844f1ee7176f53a2ff9f9081ba9

Hashes identify private source materials. Raw Sessions are not included; hashes alone cannot reproduce business scoring.

Real App execution · Independent verification · Traceable usageHistorical evidence across multiple batches / Metric definitions and limits

Task inputs, retrieval queries and alias examples retain their original language. Downloads preserve the original evaluation data.