ANAN EVALUATION · EVIDENCE REPORT

Preserve, compact or rebuild context

Original / compact / rebuild. C01–C10 and 300K context are separate; C05 has not been run.

Evidence snapshot · 2026-10-17

Continue original · original

Continue the task with the full conversation history, without actively compacting or rebuilding it.

Summary compaction · compact

The same model summarizes history, then continues with the summary and retained recent messages. Preparation requests count toward cost.

Context rebuilding · rebuild

Reassemble current resource context, recent verbatim messages and bounded history samples without a model-generated history summary. Tasks verify whether current files or memory restore older facts.

01Context strategies: outcomes and cost together

Each task instance lists selected successes / submissions, unsubmitted attempts and individual statuses. Standard and 300K context are separate. Click a run to locate its evidence.

COLD-START CACHE SCENARIO

Start cold. Compare the same tasks.

The first request starts without cache. Later requests reuse the sent prefix; new or replaced context is uncached. Compaction preparation is included.

Cache-scenario estimate, not a bill. Each chart uses its own zero-based scale.

~40–50K history context

9 tasks · 9 runs per strategy

Uncached inputMean tokens · equal task weight

Original60,502

Compact69,689

Rebuild26,462

Cached inputMean tokens · equal task weight

Original190,049

Compact116,489

Rebuild82,064

OutputMean tokens · equal task weight

Original1,219

Compact2,884

Rebuild1,202

Original · passed 8/9Compact · passed 6/9Rebuild · passed 8/9

300K context

2 tasks · 2 runs per strategy

Uncached inputMean tokens · equal task weight

Original311,983

Compact314,572

Rebuild55,348

Cached inputMean tokens · equal task weight

Original1,085,672

Compact110,432

Rebuild191,764

OutputMean tokens · equal task weight

Original3,705

Compact2,250

Rebuild1,022

Original · passed 1/2Compact · passed 1/2Rebuild · passed 2/2

Each task shows its earliest complete three-strategy cohort: one run per strategy, including failures.

Inspect a paired task

Task

Uncached inputtokens

Original52,039

Compact61,572

Rebuild16,784

Cached inputtokens

Original198,226

Compact119,028

Rebuild0

Outputtokens

Original3,555

Compact4,991

Rebuild1,893

Original · passed 0/1Compact · passed 0/1Rebuild · passed 0/1

Original · Failed5 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	48,477	48,477	0	380	48,256	First request: cold
2 · Continuation	48,927	450	48,477	74	48,768	Reuse sent prefix
3 · Continuation	49,131	204	48,927	2,489	48,896	Reuse sent prefix
4 · Continuation	51,691	2,560	49,131	321	51,328	Reuse sent prefix
5 · Continuation	52,039	348	51,691	291	51,584	Reuse sent prefix

Compact · Failed7 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Preparation	25,373	25,373	0	1,171	25,216	First request: cold
2 · Preparation	4,264	4,264	0	595	4,096	New context: cold
3 · Continuation	28,777	28,777	0	544	4,352	New context: cold
4 · Continuation	29,375	598	28,777	82	29,056	Reuse sent prefix
5 · Continuation	29,587	212	29,375	1,657	29,440	Reuse sent prefix
6 · Continuation	31,289	1,702	29,587	624	30,976	Reuse sent prefix
7 · Continuation	31,935	646	31,289	318	31,488	Reuse sent prefix

Rebuild · Failed1 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	16,784	16,784	0	1,893	4,352	First request: cold

Uncached inputtokens

Original49,535

Compact59,531

Rebuild18,004

Cached inputtokens

Original147,072

Compact87,875

Rebuild52,325

Outputtokens

Original796

Compact1,610

Rebuild955

Original · passed 1/1Compact · passed 1/1Rebuild · passed 1/1

Original · Passed4 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	48,825	48,825	0	104	25,088	First request: cold
2 · Continuation	48,991	166	48,825	213	48,768	Reuse sent prefix
3 · Continuation	49,256	265	48,991	250	49,024	Reuse sent prefix
4 · Continuation	49,535	279	49,256	229	49,280	Reuse sent prefix

Compact · Passed6 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Preparation	25,552	25,552	0	681	3,968	First request: cold
2 · Preparation	4,264	4,264	0	319	4,096	New context: cold
3 · Continuation	29,015	29,015	0	165	4,352	New context: cold
4 · Continuation	29,290	275	29,015	251	29,056	Reuse sent prefix
5 · Continuation	29,570	280	29,290	63	29,312	Reuse sent prefix
6 · Continuation	29,715	145	29,570	131	29,568	Reuse sent prefix

Rebuild · Passed4 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	17,061	17,061	0	280	4,352	First request: cold
2 · Continuation	17,451	390	17,061	293	17,280	Reuse sent prefix
3 · Continuation	17,813	362	17,451	162	17,664	Reuse sent prefix
4 · Continuation	18,004	191	17,813	220	17,792	Reuse sent prefix

Uncached inputtokens

Original49,295

Compact60,046

Rebuild18,617

Cached inputtokens

Original97,708

Compact117,015

Rebuild89,077

Outputtokens

Original794

Compact2,770

Rebuild1,894

Original · passed 1/1Compact · passed 0/1Rebuild · passed 1/1

Original · Passed3 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	48,613	48,613	0	343	48,384	First request: cold
2 · Continuation	49,095	482	48,613	172	48,896	Reuse sent prefix
3 · Continuation	49,295	200	49,095	279	49,152	Reuse sent prefix

Compact · Failed7 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Preparation	25,506	25,506	0	981	25,344	First request: cold
2 · Preparation	4,264	4,264	0	442	4,096	New context: cold
3 · Continuation	28,783	28,783	0	243	4,352	New context: cold
4 · Continuation	29,165	382	28,783	176	28,928	Reuse sent prefix
5 · Continuation	29,369	204	29,165	217	29,056	Reuse sent prefix
6 · Continuation	29,698	329	29,369	556	29,440	Reuse sent prefix
7 · Continuation	30,276	578	29,698	155	29,824	Reuse sent prefix

Rebuild · Passed6 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	17,018	17,018	0	428	4,352	First request: cold
2 · Continuation	17,541	523	17,018	227	17,280	Reuse sent prefix
3 · Continuation	17,823	282	17,541	364	17,664	Reuse sent prefix
4 · Continuation	18,235	412	17,823	197	18,048	Reuse sent prefix
5 · Continuation	18,460	225	18,235	61	18,176	Reuse sent prefix
6 · Continuation	18,617	157	18,460	617	18,432	Reuse sent prefix

Uncached inputtokens

Original49,482

Compact60,224

Rebuild18,094

Cached inputtokens

Original195,740

Compact148,393

Rebuild52,125

Outputtokens

Original877

Compact3,147

Rebuild1,111

Original · passed 1/1Compact · passed 1/1Rebuild · passed 1/1

Original · Passed5 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	48,585	48,585	0	205	24,960	First request: cold
2 · Continuation	48,864	279	48,585	64	48,512	Reuse sent prefix
3 · Continuation	49,013	149	48,864	189	48,768	Reuse sent prefix
4 · Continuation	49,278	265	49,013	177	49,024	Reuse sent prefix
5 · Continuation	49,482	204	49,278	242	49,280	Reuse sent prefix

Compact · Passed8 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Preparation	25,495	25,495	0	1,355	10,880	First request: cold
2 · Preparation	4,264	4,264	0	469	4,096	New context: cold
3 · Continuation	28,967	28,967	0	303	4,352	New context: cold
4 · Continuation	29,423	456	28,967	109	29,184	Reuse sent prefix
5 · Continuation	29,689	266	29,423	279	29,440	Reuse sent prefix
6 · Continuation	30,006	317	29,689	275	29,824	Reuse sent prefix
7 · Continuation	30,308	302	30,006	82	30,080	Reuse sent prefix
8 · Continuation	30,465	157	30,308	275	30,208	Reuse sent prefix

Rebuild · Passed4 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	16,953	16,953	0	346	4,352	First request: cold
2 · Continuation	17,445	492	16,953	118	17,280	Reuse sent prefix
3 · Continuation	17,727	282	17,445	340	17,536	Reuse sent prefix
4 · Continuation	18,094	367	17,727	307	17,792	Reuse sent prefix

Uncached inputtokens

Original49,544

Compact59,848

Rebuild18,214

Cached inputtokens

Original196,251

Compact117,681

Rebuild70,583

Outputtokens

Original704

Compact2,603

Rebuild991

Original · passed 1/1Compact · passed 1/1Rebuild · passed 1/1

Original · Passed5 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	48,632	48,632	0	214	10,880	First request: cold
2 · Continuation	49,035	403	48,632	66	48,768	Reuse sent prefix
3 · Continuation	49,184	149	49,035	189	49,024	Reuse sent prefix
4 · Continuation	49,400	216	49,184	60	49,152	Reuse sent prefix
5 · Continuation	49,544	144	49,400	175	49,408	Reuse sent prefix

Compact · Passed7 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Preparation	25,506	25,506	0	1,085	25,088	First request: cold
2 · Preparation	4,264	4,264	0	355	4,096	New context: cold
3 · Continuation	28,798	28,798	0	367	4,352	New context: cold
4 · Continuation	29,354	556	28,798	157	29,056	Reuse sent prefix
5 · Continuation	29,594	240	29,354	314	29,440	Reuse sent prefix
6 · Continuation	29,935	341	29,594	59	29,696	Reuse sent prefix
7 · Continuation	30,078	143	29,935	266	29,824	Reuse sent prefix

Rebuild · Passed5 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	17,067	17,067	0	393	4,352	First request: cold
2 · Continuation	17,649	582	17,067	65	17,408	Reuse sent prefix
3 · Continuation	17,797	148	17,649	246	17,536	Reuse sent prefix
4 · Continuation	18,070	273	17,797	60	17,920	Reuse sent prefix
5 · Continuation	18,214	144	18,070	227	18,048	Reuse sent prefix

Uncached inputtokens

Original50,472

Compact61,113

Rebuild19,435

Cached inputtokens

Original99,373

Compact120,497

Rebuild75,379

Outputtokens

Original1,055

Compact2,795

Rebuild1,310

Original · passed 1/1Compact · passed 1/1Rebuild · passed 1/1

Original · Passed3 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	49,080	49,080	0	649	10,880	First request: cold
2 · Continuation	50,293	1,213	49,080	150	49,664	Reuse sent prefix
3 · Continuation	50,472	179	50,293	256	50,176	Reuse sent prefix

Compact · Passed7 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Preparation	25,968	25,968	0	863	25,472	First request: cold
2 · Preparation	4,264	4,264	0	299	4,096	New context: cold
3 · Continuation	29,042	29,042	0	595	4,352	New context: cold
4 · Continuation	30,125	1,083	29,042	384	29,568	Reuse sent prefix
5 · Continuation	30,589	464	30,125	123	30,336	Reuse sent prefix
6 · Continuation	30,741	152	30,589	62	30,592	Reuse sent prefix
7 · Continuation	30,881	140	30,741	469	30,720	Reuse sent prefix

Rebuild · Passed5 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	17,886	17,886	0	649	4,352	First request: cold
2 · Continuation	19,023	1,137	17,886	67	18,432	Reuse sent prefix
3 · Continuation	19,170	147	19,023	101	18,944	Reuse sent prefix
4 · Continuation	19,300	130	19,170	57	19,072	Reuse sent prefix
5 · Continuation	19,435	135	19,300	436	19,200	Reuse sent prefix

Uncached inputtokens

Original50,056

Compact59,884

Rebuild18,101

Cached inputtokens

Original197,879

Compact117,473

Rebuild70,382

Outputtokens

Original984

Compact2,753

Rebuild737

Original · passed 1/1Compact · passed 1/1Rebuild · passed 1/1

Original · Passed5 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	48,855	48,855	0	340	10,880	First request: cold
2 · Continuation	49,406	551	48,855	127	49,152	Reuse sent prefix
3 · Continuation	49,700	294	49,406	191	49,408	Reuse sent prefix
4 · Continuation	49,918	218	49,700	55	49,664	Reuse sent prefix
5 · Continuation	50,056	138	49,918	271	49,920	Reuse sent prefix

Compact · Passed7 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Preparation	25,733	25,733	0	1,532	25,216	First request: cold
2 · Preparation	4,264	4,264	0	434	4,096	New context: cold
3 · Continuation	28,967	28,967	0	165	4,352	New context: cold
4 · Continuation	29,226	259	28,967	127	29,056	Reuse sent prefix
5 · Continuation	29,531	305	29,226	191	29,312	Reuse sent prefix
6 · Continuation	29,749	218	29,531	55	29,440	Reuse sent prefix
7 · Continuation	29,887	138	29,749	249	29,696	Reuse sent prefix

Rebuild · Passed5 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	17,275	17,275	0	150	4,352	First request: cold
2 · Continuation	17,512	237	17,275	93	17,280	Reuse sent prefix
3 · Continuation	17,699	187	17,512	102	17,536	Reuse sent prefix
4 · Continuation	17,896	197	17,699	178	17,664	Reuse sent prefix
5 · Continuation	18,101	205	17,896	214	17,792	Reuse sent prefix

Uncached inputtokens

Original96,522

Compact101,600

Rebuild54,923

Cached inputtokens

Original288,427

Compact43,376

Rebuild163,460

Outputtokens

Original607

Compact2,273

Rebuild747

Original · passed 1/1Compact · passed 0/1Rebuild · passed 1/1

Original · Passed4 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	95,887	95,887	0	273	4,352	First request: cold
2 · Continuation	96,189	302	95,887	134	96,000	Reuse sent prefix
3 · Continuation	96,351	162	96,189	60	96,128	Reuse sent prefix
4 · Continuation	96,522	171	96,351	140	96,384	Reuse sent prefix

Compact · Failed3 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Preparation	57,992	57,992	0	1,477	57,472	First request: cold
2 · Continuation	43,376	43,376	0	203	4,352	New context: cold
3 · Continuation	43,608	232	43,376	593	43,392	Reuse sent prefix

Rebuild · Passed4 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	54,185	54,185	0	305	4,352	First request: cold
2 · Continuation	54,519	334	54,185	209	54,272	Reuse sent prefix
3 · Continuation	54,756	237	54,519	56	54,400	Reuse sent prefix
4 · Continuation	54,923	167	54,756	177	54,784	Reuse sent prefix

Uncached inputtokens

Original97,577

Compact103,379

Rebuild55,984

Cached inputtokens

Original289,766

Compact177,066

Rebuild165,243

Outputtokens

Original1,601

Compact3,015

Rebuild1,181

Original · passed 1/1Compact · passed 1/1Rebuild · passed 1/1

Original · Passed4 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	96,208	96,208	0	344	49,920	First request: cold
2 · Continuation	96,604	396	96,208	57	96,384	Reuse sent prefix
3 · Continuation	96,954	350	96,604	595	96,640	Reuse sent prefix
4 · Continuation	97,577	623	96,954	605	97,280	Reuse sent prefix

Compact · Passed6 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Preparation	58,100	58,100	0	1,145	57,600	First request: cold
2 · Continuation	43,544	43,544	0	416	4,352	New context: cold
3 · Continuation	44,028	484	43,544	66	43,776	Reuse sent prefix
4 · Continuation	44,387	359	44,028	692	44,032	Reuse sent prefix
5 · Continuation	45,107	720	44,387	60	44,800	Reuse sent prefix
6 · Continuation	45,279	172	45,107	636	45,056	Reuse sent prefix

Rebuild · Passed4 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	54,506	54,506	0	139	4,352	First request: cold
2 · Continuation	54,986	480	54,506	737	54,528	Reuse sent prefix
3 · Continuation	55,751	765	54,986	121	55,424	Reuse sent prefix
4 · Continuation	55,984	233	55,751	184	55,808	Reuse sent prefix

Uncached inputtokens

Original313,785

Compact313,641

Rebuild54,853

Cached inputtokens

Original935,001

Compact43,196

Rebuild163,388

Outputtokens

Original5,788

Compact1,426

Rebuild708

Original · passed 0/1Compact · passed 0/1Rebuild · passed 1/1

Original · Failed4 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	308,038	308,038	0	5,285	4,352	First request: cold
2 · Continuation	313,352	5,314	308,038	231	307,968	Reuse sent prefix
3 · Continuation	313,611	259	313,352	60	313,472	Reuse sent prefix
4 · Continuation	313,785	174	313,611	212	313,600	Reuse sent prefix

Compact · Failed3 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Preparation	270,143	270,143	0	501	269,696	First request: cold
2 · Continuation	43,196	43,196	0	257	4,352	New context: cold
3 · Continuation	43,498	302	43,196	668	43,264	Reuse sent prefix

Rebuild · Passed4 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	54,185	54,185	0	307	4,352	First request: cold
2 · Continuation	54,521	336	54,185	133	54,272	Reuse sent prefix
3 · Continuation	54,682	161	54,521	60	54,528	Reuse sent prefix
4 · Continuation	54,853	171	54,682	208	54,656	Reuse sent prefix

Uncached inputtokens

Original310,181

Compact315,502

Rebuild55,842

Cached inputtokens

Original1,236,344

Compact177,668

Rebuild220,140

Outputtokens

Original1,622

Compact3,073

Rebuild1,337

Original · passed 1/1Compact · passed 1/1Rebuild · passed 1/1

Original · Passed5 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	308,359	308,359	0	356	308,224	First request: cold
2 · Continuation	308,809	450	308,359	61	308,608	Reuse sent prefix
3 · Continuation	309,163	354	308,809	822	308,864	Reuse sent prefix
4 · Continuation	310,013	850	309,163	56	309,632	Reuse sent prefix
5 · Continuation	310,181	168	310,013	327	310,016	Reuse sent prefix

Compact · Passed6 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Preparation	270,251	270,251	0	1,521	269,824	First request: cold
2 · Continuation	43,799	43,799	0	369	4,352	New context: cold
3 · Continuation	44,220	421	43,799	57	44,032	Reuse sent prefix
4 · Continuation	44,570	350	44,220	481	44,160	Reuse sent prefix
5 · Continuation	45,079	509	44,570	60	44,928	Reuse sent prefix
6 · Continuation	45,251	172	45,079	585	45,056	Reuse sent prefix

Rebuild · Passed5 requests

Original run evidence →

Request / phase	Total input	Scenario uncached	Scenario cached	Output	Observed cached	Cache rule
1 · Continuation	54,506	54,506	0	247	4,352	First request: cold
2 · Continuation	54,805	299	54,506	57	54,656	Reuse sent prefix
3 · Continuation	55,155	350	54,805	491	54,784	Reuse sent prefix
4 · Continuation	55,674	519	55,155	56	55,296	Reuse sent prefix
5 · Continuation	55,842	168	55,674	486	55,680	Reuse sent prefix

Download scenario data (JSON)

Case	Original	Compact	Rebuild
40–50K history context
C01Preserve dietary, quantity and budget facts; separate clarification from guessing	FailedTotal omitted the 2,400-cent shipping fee; 31,200 instead of 33,600.	FailedTask output correct. Automatic memory creation conflicted with the evaluator's unchanged-memory rule.	FailedRebuilt context omitted earlier requirements; asked for clarification without producing the file.
C02Recover pickup facts from current memory	Passed	Passed	Passed
C03Use current file rather than stale owner, time and budget	Passed	FailedTask output correct. Automatic memory creation conflicted with the evaluator's unchanged-memory rule.	Passed
C04Process remaining work and protect completed deliverables	Passed	Passed	Passed
C06Integrate address, date and fee revisions	Passed	Passed	Passed
C07Combine valid receipts and reversals under current policy	Passed	Passed	Passed
C08Skip completed work; include failed/reopened work	Passed	Passed	Passed
C09Recover eight fields for one booking; never guess the code	Passed	FailedSummary omitted the booking's exact fields; requested the original record without producing the file.	Passed
C10Merge approved revisions and select a feasible current option	Passed	Passed	Passed
300K context
C09Recover eight fields for one booking; never guess the code	FailedOriginal record was available, but the budget and confirmation code were copied incorrectly.	FailedSummary omitted the booking's exact fields; requested the original record without producing the file.	Passed
C10Merge approved revisions and select a feasible current option	Passed	Passed	Passed

Strict passes / submissions includes selected repeats; unsubmitted attempts are listed separately. Meeting the goal does not replace a strict pass. Successful-run means require complete usage; failures and missing data remain in run details. ≥ indicates a known subtotal. Complete fields do not guarantee that every historical request was observable.

A small set of manual cases and historical repeats cannot prove unchanged overall accuracy or stable benefits. Fewer total tokens do not imply proportional monetary savings: caching, preparation requests and tool routes affect cost. C05 has no real submission under the current definition. A historical C09 batch exceeded its budget; budget failures and task grades remain separate.

Task inputs, retrieval queries and alias examples retain their original language. Downloads preserve the original evaluation data.