Model Routing and Fallback Systems
Model routing with an LLM router: cheap versus strong pins, canaries, eval-based routes, version pins, and when a fallback model should take over a request.
- Read time
- 15 min
- Level
- Intermediate
- Updated
- 2026-08-25
- Components
- Request classifier · Model catalog · Canary lane
A model routing system is an LLM router that sends each request to a cheap, fast, or strong model based on task class and eval scores, pins versions, canaries changes, and fails over along a declared fallback chain when a provider errors, times out, or fails a quality gate.
Key takeaways
- 01
Route by task difficulty and risk, not by a single default frontier model; extraction and classification rarely need the expensive tier.
- 02
Pin model versions. Unpinned "latest" is an unscheduled deploy of someone else's weights.
- 03
Fallbacks are a chain with the same schema contract — not a silent swap to a model that cannot call your tools.
- 04
Canary a percentage of traffic against the golden set before promoting a new pin; eval-based routing beats gut feel.
- 05
If every request is high-stakes or the catalog has one model, you do not need a router — you need a pin and a health check.
Model routing: cheap, strong, and fallback
Model routing is the control plane that chooses which model handles a call. An LLM router classifies the request (or the agent step), looks up a catalog of pinned models with cost, latency, context, and tool-support flags, and dispatches. A fallback model is the next catalog entry when the primary errors, exceeds SLO, or fails a cheap quality check.
The point is not cleverness. It is to spend frontier tokens on planning, judgment, and hard generation, and to spend small-model tokens on extraction, classification, rewriting, and judges. Declarative routes — task type → model pin → fallback list — are testable. A prompt that says "pick a model" is not.
- Classifier or explicit task-type from the orchestrator
- Catalog: pin, context window, tools, cost, region
- Canary lane with eval comparison
- Fallback chain with schema and tool compatibility
The catalog, pins, and compatibility
The catalog is the source of truth: provider, model ID, exact version or snapshot, region, max tokens, structured-output support, tool-calling support, and data-handling terms. Agents and routers reference aliases ("planner-strong", "extract-fast") that resolve to pins. Changing a pin is a deploy.
Compatibility matters more than raw quality. A fallback that cannot emit your JSON schema or cannot call tools is not a fallback; it is a different product. Record capability flags and refuse to route a tool-using span to a completion-only model.
Eval-based routing and canaries
Choose routes with evidence. Offline, run the golden set on each candidate pin and keep a route only if it meets the quality bar for that stratum at a cost and latency you accept. Online, canary a few percent of traffic to a candidate pin, compare judge scores and hard graders, and promote or roll back.
Some routers score difficulty live (small classifier, or a cheap model that may escalate). Escalation must be bounded: one upgrade per request is typical. Open-ended "keep trying bigger models" is how a ten-cent request becomes a dollar without a better answer.
Fallback chains and health
A fallback chain is ordered: pin A → pin B (same capability class) → degrade (cached answer, extractive snippet, or a typed error). Trigger on timeouts, 5xx, rate limits, and optionally on schema-invalid output after retries. Do not fallback on 4xx that indicate a bad request — that loops.
Health of a provider is an input to the router, not a surprise. If error rate on pin A crosses a threshold, shed new traffic to pin B and page. User-visible traces should show which pin served the request so support is not debugging the wrong model.
Failure modes and when not to route
Classic failures: unpinned aliases that change overnight; a fallback that drops tool support; a classifier that sends hard legal questions to the cheap tier; canaries without a rollback; routers that retry four models and bill all of them. Cost dashboards that ignore fallbacks under-count.
Skip a router when you have one qualified model, or when every request is in the same high-risk class (then pin the strong model and add health-check failover only). Also skip live difficulty classifiers you have not eval'd — a misrouted 5% of traffic is a quality incident that looks like a model problem.
- Unpinned "latest" versions
- Incapable fallbacks (no tools, no schema)
- Unbounded escalation and retry billing
- Classifier errors on high-risk strata
Operating the router
Treat route tables like code: review, CI against the golden set, canary, then promote. Dashboard cost per task class, fallback rate, and per-pin quality. Pin versions on a calendar so provider deprecations are planned. Work stays model-agnostic — Anthropic, OpenAI, Google, Mistral, open-source, or a client fine-tune — because the catalog, not the app, names the weights.