Lesson

Measure the endpoint, not the proxy

Score the thing the reward scores, at the tier the reader cares about, because aggregates hide dissociation.

The incident

Two training runs were compared at “equal steps”. They had not seen equal text; the byte count per step was 21% low on a half-code corpus, so the comparison was between different amounts of training. Elsewhere, a benchmark reported one aggregate number across tiers whose per-tier results moved in opposite directions. The aggregate said “no change”.

A thinking-budget parameter was found to be charged per round trip, so two serving stacks that batch tool calls differently spend different budgets on the same task. The comparison across stacks was not a comparison.

The rule, and how it is enforced now

• Comparisons are made at equal text, never equal steps.
• Every benchmark reports every tier. A single aggregate is not accepted as a result.
• Anything charged per turn is compared within one stack, or reported with turn counts.
• Verdicts never source their evidence from the component under test.