Local or cloud

Why not send everything to a frontier model? Because a fleet of agents makes thousands of calls a day, most of them small, and three things differ by an order of magnitude between a local model and a frontier one: the price per call, what the output may legally be used for afterwards, and how much judgement the call actually needs. The router is where those three get decided per workload instead of per conversation.

The shape today

Solid lines are the normal path. Dotted lines are the configured fallbacks, and the point of drawing them is that almost every one ends in the paid tier. That is a resilience feature with a bill attached, and it is the subject of the first trap below.¹

Era 1: Bifrost, one profile per agent (2026-06)

Dream Team’s agents talked to models through a profile-router fork of Bifrost, an open-source LLM gateway. Each agent got a profile and a virtual key. A profile was a tiered list: tier 1 a local model on monte (a 30B coder on vLLM for builders, a small Gemma on Ollama for the PM), tier 2 a cloud fallback (Gemini Flash) if the local tier failed. Builder sessions were sticky to one backend so a half-finished job did not change models mid-thought.²

The builder pool was itself tiered, lite / medium / heavy, from cheap local models up to frontier cloud models, matched to task size. A 33-minute A/B soak in May put numbers on it: one cloud model delivered 3 of 4 specs for $1.81, another delivered 4 of 4 for $10.26, about 5.7× the total and 3 to 4× per useful task, with retry and no-op cycles eating half the spend on both. The finding that mattered was not which model won; it was that retry loops, not model choice, were the cost, and that three of the bugs behind them were runtime gaps that would bite any high-throughput pool.³

Two more controls lived beside the router: a cheap-model wake filter that decided whether the expensive main process should wake at all, and a daily dollar budget per agent that paused it on a quota error. Both are ancestors of everything below.⁴

Era 2: LiteLLM, one profile per workload (2026-07 → now)

The greenfield platform replaced per-agent routing with a LiteLLM gateway whose profiles are named for the job, not the model. Consumers ask for local-coder or security-review; what serves it can change under them.

Assignments were made by measurement, not preference: the 35B judge scored 100% against a smaller model’s 85.7% on the spec-lint task, and wrote correct tests 3 of 3 times where the larger heavy model managed 1 of 3, so each got the job it was good at. Moving security review from a cloud profile to a local one closed the platform’s first cloud-spend priority-zero. Cloud spend got a dashboard the same week the platform cut over.⁶

The rules, and why

1. Local first, then DeepSeek or GLM, and no Anthropic in fleet tooling. Set 2026-07-17. Two reasons. Licensing: the fleet’s harvest loop turns model output into training data for small models, DeepSeek’s terms permit distillation, GLM is open-weight-adjacent, and Anthropic’s output cannot feed that loop. Cost: a flash-class run is about $0.004 against $0.10 to $0.24 for a frontier model on the same task. Claude Code is where I work and where oversight sessions run; it is not a component the fleet calls.⁷
2. Escalation is plumbing, and the small model should learn to ask. The long-term shape is an outer tier of frontier cloud vendors and an inner tier of local models, search and retrieval, with escalation trained into the small models as a priced action rather than only gated outside them. Harvested escalations are, by construction, the hard examples the small model should be trained on next.⁸
3. A profile names a job. Swapping the model behind local-coder is a config change and a benchmark, not a code change across every caller.
4. Fallbacks are a resilience feature with a hidden bill. Every local profile falls back to the cloud judgement tier. That is deliberate, and it means a dead local backend turns into paid, off-site calls that return a perfectly normal 200.

Hooks into the bench lane

The router only works if the assignments stay true, and they only stay true if someone keeps measuring. That is the bench lane’s job, and three of its findings feed straight back here:

• Per-tier results, never one number. A quantisation that costs nothing on prose and 15 points on code is fine behind local-utility and wrong behind local-coder. The lane reports every tier so a profile can be assigned on the tier it will actually serve.
• Thinking budget is charged per round trip, not per task. Two serving stacks that batch tool calls differently spend different budgets on the same job, so a profile’s cost must be compared within one stack or reported with turn counts. Comparing across stacks on that axis is not a comparison. ⁹
• A prompt cache lives in host RAM. About 2.2 GiB per saved 32k-token slot on one serving engine, which is how a bench box ran out of memory. The router’s local tier has a memory budget as well as a GPU budget. ¹⁰

Assignments made from bench results so far: the 35B judge over the 3B model for spec-lint (100% vs 85.7%), the markov model over the heavy model for test authoring (3 of 3 vs 1 of 3), and the community 4-bit build of the heavy model retired after the SnapDragon-Coder ablation showed the quantisation, not the model family, was the quality killer.

Two traps worth publishing

• The alias lies; the response does not. In August a local profile’s backend was down and every call quietly came back from a paid cloud model, as configured. The only evidence was the model field in each response. Rule since: check what answered, never what was asked for. Today that is a habit in the harness code that reads the response, not an assertion that fails the call; making it one is on the list, and the site will say when it is. ¹¹
• A fallback group that ends in the cloud ends when the balance does. On 2026-08-24 the DeepSeek account ran dry overnight. Every call through the cloud tier returned 402 with no further fallback, and a data build “completed” thousands of rows short before anyone noticed. Local lanes were untouched. Only I can top it up. ¹²

What does not go through the router

Training runs talk to no gateway. Benchmarks hit serving stacks directly, because the router would hide exactly the per-stack differences the bench lane measures. And the read-tuning data for Marilyn’s experts was generated and judged by a frontier model reached directly, under the same licensing test as rule 1.¹³