How to Reduce LLM API Cost in Production
How to reduce LLM API cost in production: routing, prompt caching, semantic cache, smaller models, batching, and CI evals so quality does not silently drop.
To reduce LLM API cost in production, route easy requests to smaller models, cache identical and similar prompts, shrink only the context you have measured as unused, batch offline work, and gate every change with a golden-set eval so quality does not silently drop.
What you’ll build
- A cost dashboard that splits spend by route, cache hit, retry, and eval traffic
- A router that sends easy turns to a small model and hard turns to a frontier model
- Exact prompt caching plus a semantic cache with a measured hit-rate and TTL
- Offline batch jobs moved off the interactive endpoint
- A CI quality gate so a cheaper route cannot ship if the golden set drops
Before you start
- 01Itemized provider invoices with input, output, cached, and tool-call tokens
- 02Trace IDs on every production completion, including retries and judge calls
- 03A golden set of at least 100 real tasks with pass/fail rubrics
- 04Pin-able model IDs (no silent default-model drift)
- 05A quality floor you will not cross to save money
Key takeaways
- 01
Most production bills are context tokens, retries, and hidden judge calls — not the headline output token price.
- 02
Prompt caching pays when the system prefix is long and stable; a semantic cache pays when users repeat paraphrases of the same question.
- 03
A smaller model on easy turns plus a frontier model on hard turns beats one mid-size model on everything, but only if the router is scored against the same golden set.
- 04
Batch APIs and async queues belong to reports, backfills, and overnight evals — not to interactive chat.
- 05
Never ship a cost change that you cannot revert by flipping a route flag; pin model versions and keep the previous route warm.
How to reduce LLM cost starts with the invoice, not the prompt
Teams that try to reduce LLM cost by rewriting one system prompt usually miss the real line items. Production spend is input context, retries after JSON failures, tool loops, embeddings, and the LLM-as-judge you bolted on last quarter. Pull traces for a week and group them by surface: chat, agents, batch extractors, and eval jobs. You cannot cut an OpenAI bill you have not attributed.
Set one metric the finance partner will accept: dollars per successful task, with success defined by the golden set or a deterministic checker. Token price is a vendor input. Task cost is the number you can manage. Publish it weekly next to quality so a cheaper route that fails users is visible as a cost increase, not a saving.
Route easy turns away from frontier models
A production mix is not one model. Classification, short FAQ turns, and structured extraction often pass on a small or distilled model. Multi-hop tool use, messy policy questions, and anything that writes to a system of record still need a stronger model. Build a router that scores difficulty from the user text, retrieved-chunk count, and tool schema — then send traffic accordingly.
Score the router itself. Log which model answered, whether the answer passed the rubric, and how often the small model was escalated. If escalation exceeds your budget, the classifier is guessing. Start with rules (length, presence of an account ID, tool-write intent) before you train a learned router. Pin every destination model ID so a provider update cannot reprice the fleet.
- Easy: FAQ, classification, extraction with a tight schema
- Hard: multi-tool agents, long documents, ambiguous policy
- Write path: stronger model plus confirmation, never the cheap route
- Unknown: fail closed to the strong model or a human, not to silence
Prompt caching and semantic cache
Prompt caching (the provider feature) reuses a long, identical prefix: system instructions, tool JSON, and static few-shots. Put those bytes first, keep them byte-stable, and you pay full price once per cache window instead of on every turn. It does nothing for unique user text. If your prefix is short, caching will not cut the bill; shrink tools and retrieval instead.
A semantic cache sits in front of the model: embed the request, look up a near-duplicate for the same tenant and corpus version, and return the stored answer if similarity and freshness pass. This is how you cut paraphrase traffic — 'reset laptop' and 'how do I factory reset my notebook' — without hoping the provider prefix cache notices. Key by tenant, locale, and index version. Never cache tool-write results or answers that include another user's records.
Smaller models, shorter context, same rubric
Smaller models are cheaper per token and faster, which also cuts timeout retries. They fail more on long-context reasoning and tool argument shape. Do not swap the fleet in one deploy. Shadow the small model on a slice of traffic, grade it on the golden set, and only then move easy routes. If the small model needs a 4k-token scratchpad to match the large model, you have not saved money.
Context is the other half. Every unused tool schema and every retrieved chunk you never cite is billed input. Measure citation rate per chunk position and drop the tail. Prefer summaries of old conversation over raw history. Do not strip citations or policy clauses to hit a token budget — that is a quality cut dressed as a cost project.
Batching and async jobs
Interactive chat should not share a queue with overnight document extraction. Providers bill batch and reserved capacity differently; even when the unit price is similar, batching removes peak-time contention and retry storms. Move embeddings, re-ranking backfills, transcript summaries, and scheduled evals to an async worker with idempotent job IDs.
Inside a single request, batch tool results and judge calls. Five separate 'is this faithful?' completions cost more than one structured judge over five claims. Cap agent loops. A tool-calling agent that retries a flaky API eight times is a cost incident, not a robustness feature. Timeouts and circuit breakers belong in the same review as the model invoice.
Evals so quality does not silently drop
Every cost change is a model change. Run the same golden set before and after routing, caching, or truncation. Track retrieval hit rate, task pass rate, and groundedness — not only spend. If pass rate falls by more than your agreed floor, roll back. Cheap answers that dump work on support are not cheaper.
ReinforcedX implements this as a four-week production change inside your cloud: week 1 attributes the bill and freezes the quality floor, week 2 wires routing and caches, week 3 runs a shadow-mode comparison, week 4 hands over dashboards and runbooks. You own the evals and traces. There is no token markup; you pay the provider. Thirty days of on-call after handover is included so a cache miss storm is not your first production surprise.
Step-by-step build
- 1
Attribute the current bill
Break last month's invoice into input, output, cached, tool, embedding, and eval tokens, then join traces to product surfaces so you know which workflow actually spends the money.
- 2
Pin models and freeze a quality floor
Stop using provider aliases, record the current golden-set pass rate, and treat that number as the floor any cheaper route must match before it can take traffic.
- 3
Install a model router
Classify each request as easy, hard, or unsafe; send easy turns to a small model, hard turns to a stronger model, and fail closed to a human or refusal when the classifier is unsure.
- 4
Add prompt caching and a semantic cache
Place the stable system prompt and tool schemas first so the provider cache can reuse the prefix; add a tenant-keyed semantic cache with embedding similarity, TTL, and corpus-version invalidation.
- 5
Trim context and move batch work off the hot path
Drop unused tools, cap retrieved chunks by measured contribution, and send reports, re-embeddings, and nightly judges through a batch or reserved-capacity endpoint.
- 6
Gate cost changes with evals
Run the golden set on every routing or cache change, alert when cost per successful task rises or pass rate falls, and keep a one-flag rollback to the previous route.
Common pitfalls
The mistakes that show up in real deployments — each one costs a week if you learn it the hard way.
Optimizing dollars per token instead of dollars per successful task
A cheaper completion that retries twice, calls a judge, and still fails the user costs more than a single correct call. Track cost per passed task, not cost per million tokens.
Turning on a semantic cache with no TTL or tenant key
Paraphrase caches leak answers across users and go stale the moment a policy doc changes. Key the cache by tenant plus corpus version, and expire it when source documents update.
Shrinking the prompt until retrieval context disappears
Aggressive truncation is a silent quality cut. Measure groundedness after every context trim; if hit rate falls, you did not reduce cost — you moved the bill into human cleanup.
Forgetting that evals, traces, and retries are billable
LLM-as-judge on 100% of traffic can dwarf the original completion cost. Sample judges, reuse traces, and never log full prompts to a second paid model without a budget line.
Leaving model aliases unpinned
Provider 'latest' aliases change weights under you. Pin versions, canary a new snapshot against the golden set, then switch. Unpinned aliases undo weeks of routing work overnight.