LLM Cost FAQ
Token economics, the hidden multipliers, and the controls that cut spend without silently dropping quality.
LLM cost is mostly billed tokens — input and output, at different rates — multiplied by retries, long RAG context, judges, and agent loops, which is why production bills exceed the playground. You reduce it with caching, smaller models on easy cases, tighter prompts, and evals so quality does not fall while the invoice does; clients pay providers directly, with no token markup from us.
How LLM cost actually works
If you remember one mechanism: you pay for every token in and every token out, and production sends far more of both than a demo.
How much does the ChatGPT API cost?
It is billed per token, with separate rates for input and output, and extra for features such as cached input or higher-tier models — list prices change, so we do not quote them here. Your invoice is rate × tokens, plus any reserved capacity. Consumer ChatGPT Plus is a different product with a seat fee, not an API meter. Check the provider’s current page for numbers; design for the mechanism so a price change is a config, not a panic.
What are tokens, and why do they matter for cost?
Tokens are sub-word chunks the model reads and writes. English often lands around a few characters per token; code and other languages differ. Billing is on token counts, not words or API calls. A long RAG prompt can cost more than the sentence it produces. Counting tokens in traces is how you attribute spend to a workflow instead of staring at a monthly lump.
Why are input and output tokens priced differently?
Filling the context window (input) and generating new tokens (output) stress the hardware differently; providers set separate rates, and output is commonly more expensive per token. Reasoning or “thinking” tokens, when billed, behave like output. That is why a terse answer with a huge retrieved context is an input problem, and why chain-of-thought on every call is an output problem. Optimise the side the traces say is large.
Why is the production bill higher than the playground?
The playground hides the system prompt, does not retrieve 20 chunks, does not retry on JSON failures, does not call a judge, and does not loop an agent five times. Each of those is a full model call or a pile of extra input tokens. Logging and evals add more. Forecasting from a single happy-path completion is how finance gets a surprise in month two.
How do you reduce LLM API cost in production?
Route easy cases to a smaller or cheaper model, cache repeated prefixes and frequent questions, shrink RAG context with better retrieval instead of stuffing, cap agent steps, batch where latency allows, and stop retrying blindly. Every change runs the eval suite so quality cannot drop silently. The how-to on reducing LLM API cost is the playbook. Cutting the model first without a gate is how you save money and lose the workflow.
Do prompt caching and semantic caches actually help?
Yes, on repeat. Provider prompt caches discount long, stable prefixes (system prompts, tool schemas). A semantic cache returns a stored answer when a new question is close enough to an old one — with a similarity threshold and a TTL, and never on write actions. Caches that skip the eval’d path on novel questions will serve stale policy. Hit rate is a metric; it is not a reason to skip grounding.
How teams actually take cost down
The order matters: measure, cache, route, then shrink. Skipping to “smaller model” is how quality disappears.
Does a bigger model always cost more in production?
Per token, usually yes. Per successful task, not always: a stronger model may use fewer retries, shorter plans, or skip a judge. A cheap model that loops six times can lose. Routing is how you get both — small model when the classifier says the intent is easy, large when it is not. The eval suite, not the nameplate, decides which path wins on a slice.
How does model routing save money?
A router sends each request to the cheapest model that still passes the eval for that intent, with a fallback if confidence is low or the cheap path fails schema checks. Canaries and version pins stop a silent update from flipping the mix. Routing without per-intent scores will send hard tickets to a small model and you will pay in escalations. You still pay each provider directly.
When does batch inference beat realtime?
When the user is not waiting: overnight extraction, eval suites, backfills, report generation. Batch APIs and queues trade latency for a lower unit cost and higher utilisation. Customer chat, voice, and agent tools that block a UI stay realtime. Mixing them on one hot path is how you either overpay or miss an SLO. We split the two in the architecture, not in a spreadsheet after the fact.
Who pays the model provider — you or us?
You do. Keys and billing live in your account. ReinforcedX never resells tokens with a markup. Our commercial model is a platform subscription plus a fixed-scope implementation fee, quoted in writing. If a self-hosted model is the path, you pay the cloud GPUs. This is deliberate: your rates, your commitments, your ability to leave.
How do you forecast cost before going live?
From traces in shadow mode: tokens in/out per intent, retry rate, retrieval size, judge sampling rate, and expected volume. Multiply by current provider rates (which you look up; we do not freeze them in a PDF). Add a buffer for retries and incident reruns. A spreadsheet with “one completion per ticket” is fiction. Week three of the four-week implementation is when this number becomes real.
What does a four-week implementation cost versus inference?
They are different bills. Implementation is a fixed-scope fee for the system we hand over. Inference is the ongoing token or GPU meter you pay the provider. We will not bundle them into a blended “AI tax.” A well-built system makes inference visible per workflow so finance can compare it to the human minutes it displaces. If volume is tiny, inference will look small next to build; if volume is huge, the reverse.
What do you need from us to attack the bill?
A week of traces with token counts, the current model list, volume by intent, and someone who can change routing config. If you have no traces, week one is instrumentation, not discounts. We will not guess a 40% cut from a screenshot of the playground.
Keeping the bill under control once you are live
Runaway loops and silent model updates are how a stable month becomes a bad invoice. Caps and evals are the fix.
How do you stop a runaway agent bill?
Hard caps: max steps, max tokens per run, max concurrent runs, and a kill switch on the tool gateway. Budget alerts on daily spend per workflow, not only on the corporate account at month end. Infinite retry on a failing tool is the classic incident. These caps are in the runtime, not in a Slack reminder. Shadow mode is where you learn the distribution before write tools open.
Won’t caching or a smaller model hurt quality?
They can, which is why both sit behind the golden set and a live sample. Semantic caches need a similarity floor and a freshness TTL; they are wrong for anything that can change state. Smaller models get the intents they already pass. If the suite drops, the change rolls back. Cost work without evals is how you invent a cheaper, worse product.
Is self-hosting cheaper than APIs?
It can be at steady, high utilisation on open-weight models, and it can be far more expensive at spiky or low volume once you count GPUs, idle time, and people. APIs win on burst and on frontier quality. Many production stacks mix: API for hard cases, self-host for the bulk. We model utilisation and residency together; we do not have a religious answer. You pay whichever meter you choose.
How much cost do evals add?
Offline suites are batch: real, but not in the user path. Online judges sampled on a slice of live traffic are the line item people forget — a second (or third) model call. Tune sample rate by risk. Rubric checks that are deterministic (schema, citation present) are nearly free. Dropping evals to save judge tokens is how you fail the next silent update. Put judge spend on the same dashboard as completions.
Should we count hallucinations and retries as cost?
Yes. A wrong answer that triggers a retry, a human take-over, or a refund is spend, even if it is not an API line. Tracing those events next to token counts is how you decide whether a grounding pass pays for itself. Cheap, ungrounded completions are not cheap. Finance and quality should share a dashboard or they will fight with different numbers.
How do you attribute LLM cost per workflow?
Tag every call with workflow, intent, model, and tenant in traces, then sum tokens and vendor dollars by those tags. Shared system prompts still belong to the caller. Without tags you get one invoice and an argument. Observability setup is part of implementation, not a later “nice to have.” You own the traces at handover.
When cheaper is the wrong call
Cost is a constraint. It is not the objective on irreversible actions.
When is a cheaper model the wrong call?
When the slice is high-stakes (money, health, legal), when evals already fail the small model, or when retries would erase the saving. Also when latency from loops would miss an SLO. Pin a stronger model there and save money on the bulk intents. Across-the-board downgrades are how a “cost project” becomes an incident project.
How does cost differ for support vs internal vs batch extraction?
Support is realtime, often RAG-heavy on input tokens, with a human fallback that hides some quality loss until it does not. Internal search can cache aggressively. Batch extraction should never pay realtime rates. Voice adds STT/TTS meters besides the LLM. Design the path to the channel; copying the support stack onto overnight PDF jobs is wasted money.
Do we need to switch vendors to save money?
Not first. Caching, routing, and context shrinkage usually move the needle on the vendor you already have. Switching families means re-tuning prompts and re-running evals. We will switch when the suite says another model wins on quality-per-dollar, not because a tweet said it was cheaper. Model-agnostic means we can; it does not mean we must.
What usually blows the budget in the first 90 days?
Unbounded agent loops, logging full traces back into the next prompt, judges on 100% of traffic, vector-search stuffing 30 chunks, and no daily alerts. The other classic is a silent upgrade to a dearer default model. Caps, sampling, and a per-workflow dashboard in week two prevent most of this. Rescue starts by reading traces, not by renegotiating the provider contract.
If we pay the provider anyway, why hire an implementation partner?
Because the spend is determined by architecture — retrieval size, retries, routing, caps — not by who clicks the API. We build those controls in your cloud in four weeks, hand over the runbook, and do not sit on the token margin. A cheaper unit price on a wasteful loop still loses. That is the job; it is not a resale of someone else’s models.
Production LLM spend is token traffic plus the systems around it: retrieval, retries, eval judges, traces, and self-hosted GPUs if you go that way. Input tokens (prompts, RAG chunks) and output tokens (completions, chain-of-thought) are priced differently; output is usually dearer per token. Hidden multipliers are retries, agent step loops, and sampling a judge. Controls are routing, prompt and semantic caches, shorter context, batch where latency allows, and step limits. ReinforcedX does not resell tokens. Quality gates travel with cost work so a cheaper path cannot ship a worse agent unnoticed.
Keep reading
Ready to refine
your workflow?
Share your current process. We’ll help you identify what can be automated — and where efficiency can be reclaimed.