Bench lane
- status
- active · ongoing
- when
- 2026-08 → now
- came from
- Marilyn, SnapDragon-Coder, First public repos
Measuring what quantisation and serving stacks actually cost a model, per tier, with the noise floor stated.
Why it started
SnapDragon-Coder died of a quantisation nobody had measured. The lab needed a way to say “this quant costs this much on this kind of task” before trusting a model on a card.
What it is
Benchmark matrices committed as specs, results committed as evidence, run on whichever machine holds the lease. Every tier is reported; an aggregate is not a result. The first programme compared a Qwen3-based coder across quantisation levels and two serving stacks.1
Where it stands
Active. Two findings so far are worth the whole lane: a serving engine’s prompt cache lives in host RAM at about 2.2 GiB per saved 32k slot, which is how a bench machine ran out of memory; and the thinking-budget parameter is charged per round trip, so stacks that batch tool calls differently are not comparable on that axis. See measure the endpoint.2
Lineage
- Came from: Marilyn, SnapDragon-Coder
- Led to: the assignments on Local or cloud. The lane is the instrument; the router is where its readings become decisions.
Sources
Numbers on this page are quoted from the lab's own records. The records are private; each note gives the record's date.