News

Model routing moves from gateways into agent harnesses

A discussion amplified by Keith Rabois on X on 16 August 2026 highlights a strategic split emerging in AI infrastructure: should model selection happen at a neutral gateway, or inside the agent harness that understands the task? The underlying argument, set out in a 15 August thread by Jerry Liu, is that every task needs a different mixture of models and an agent harness can optimise the end-to-end workflow rather than treating each completion as an isolated request.

Why the harness matters: a gateway can see the incoming prompt, provider health, policy constraints and prices. The harness additionally sees the session history, cache state, tool results, current sub-task, failed attempts, tests and delivered artefacts. That context lets it choose a model before constructing the next request, adapt the available tools and instructions to the selected model, assign specialised sub-agents, and decide whether a model switch is worth the loss of a warm prompt cache. The consequence is important: routing is not simply a price-and-latency lookup. In long-running agent sessions, a cheaper model can still be more expensive if switching forces the accumulated context to be processed at fresh-input rates.

Factory's production thesis: Factory says its Router has been used in production for more than two months and reports a 58% aggregate cost reduction against using a frontier model for every call, 76% median session savings, and median turn latency falling from 81 seconds to 49 seconds. It says routed sessions matched frontier-pinned sessions across eight production measures, including task completion, failures, repeated edits, delivered artefacts and user acceptance. These are company-reported results rather than independently audited benchmarks, but they illustrate the category's shift from outage failover to outcome-aware optimisation. Factory's earlier model-routing launch was reported on its profile as claiming a 25% cost reduction.

How the landscape is separating: OpenRouter represents the gateway and marketplace layer: its Auto Router classifies prompts, ranks models using anonymised task-spend signals, applies cost tiers and account restrictions, and supports session stickiness. Portkey.ai represents the enterprise gateway layer, combining a universal API with fallbacks, conditional routing, caching, guardrails and observability. Unify AIDealroom has a profile for this one. Try Dealroom → is another adjacent model-orchestration player in the broader market. These approaches are complementary rather than mutually exclusive: a harness can select the task-level model while a gateway enforces policy, handles provider selection and failover, and preserves a stable interface.

The investable question: as agents run for hours and generate dozens or hundreds of calls, value may migrate towards the layer that owns task decomposition and outcome data. Gateways retain structural advantages in neutrality, centralised governance and broad application coverage; harnesses retain informational advantages for multi-step work. The likely architecture is therefore a stack, not a winner-takes-all replacement: harness-level routing for task and session decisions, gateway-level routing for policy, reliability and provider execution. The strongest products will need to prove quality-adjusted cost savings on comparable tasks, preserve cache economics when switching models, and show that routing decisions improve from verified outcomes rather than merely tracking popularity.

Read more: Keith Rabois on X · Jerry Liu on X · Factory · OpenRouter documentation · Portkey.ai documentation

More top stories