Reasoning Models FAQ
Reasoning models spend extra tokens at answer time to search a problem — useful for hard steps, wasteful if you put them on every greeting.
Reasoning models (o1-class, “thinking” or test-time-compute models) generate hidden or explicit intermediate steps before the final answer, trading latency and tokens for better performance on multi-step problems. They are not a guarantee of truth, and they are the wrong default for every turn of a production agent. Use them where a cheaper model fails a named eval.
What reasoning models are
If you only need one distinction: they spend more compute per answer. That is the feature and the bill.
What are reasoning models?
Models trained or served to use extra inference-time computation — generating intermediate steps, searching, or “thinking” — before they emit the user-visible answer. OpenAI’s o-series, similar “thinking” modes from other labs, and some open-source long-CoT models sit here. They are still language models. They guess unless they are grounded in tools or retrieval. The new knob is how long they are allowed to work on one item.
How is o1 different from a regular GPT chat model?
A chat model answers in one forward pass of visible tokens. An o1-class model spends a budget of hidden or internal tokens on the problem, then answers. That helps on contest math, some coding, and multi-step puzzles. It is slower and usually more expensive per request. It is not automatically better at your policy Q&A. Compare them on your golden set, not on a public leaderboard.
What is test-time compute?
Compute spent when answering, not when training: longer rollouts, more sampled paths, or an explicit thinking budget. You can often buy quality by waiting and spending tokens instead of training a new model. The curve saturates. Past a point you pay for loops. In production the budget is a product choice — milliseconds on voice, seconds on back-office — not an unbounded “think harder” slider.
Is chain of thought the same as a reasoning model?
No. Chain of thought is a prompting pattern: ask a chat model to show steps. A reasoning model is trained or served to allocate a thinking budget, often with hidden tokens you never see. Prompted CoT is cheap to try and sometimes enough. If CoT already clears the eval, do not rent a slower SKU. If it does not, a reasoning model is one of the options, next to tools and better retrieval.
When should we use a reasoning model in an agent?
On the steps where a fast model fails a named metric: planning a tool sequence, writing non-trivial code, comparing conflicting sources, or hard classification with a large rubric. Not on greetings, slot filling, or “look up this order.” Agents that reason on every turn miss latency SLOs and burn the month’s token budget by Wednesday.
Why are reasoning models so expensive and slow?
Because you are buying many extra tokens and sequential steps per answer. Providers charge for those tokens; wall-clock time grows with the thinking budget. There is no public price we will invent here — read your contract. We pass through your rates with no markup. The way to control the bill is routing, budgets per step, and not using a reasoning model as the only model in the graph.
Do reasoning models hallucinate less?
They hallucinate differently. They are stronger on problems with a checkable answer and still willing to invent citations, policy, and tool results if you let them speak without retrieval. A fluent chain of thought is a convincing wrong answer. Grounding, refusal when retrieval is empty, and evals remain the controls. Do not drop citations because the model “thought hard.”
How we actually use them
The implementation is a router and an eval, not a wholesale model swap.
Should every agent step use a reasoning model?
No. Most steps are classification, retrieval, or a short reply. Put a reasoning model on the minority of steps your eval identifies, with a token and time budget. If you cannot name those steps, you are not ready to pay for thinking. We will instrument the cheap model first; promotion is a measured change, not a default.
Can we route easy cases to a cheap model?
Yes, and that is the production pattern. A small classifier or the fast frontier model handles the head of traffic; hard cases escalate. Canaries and pinned versions stop a silent upgrade from moving you onto a slower SKU. You pay each provider directly. We will not resell a “reasoning tier” with a markup.
How do you evaluate reasoning quality?
Same spine as any LLM eval: a frozen golden set, a rubric, automated scoring, a CI gate, and a live sample. Add cost and latency as first-class scores so a quality bump that triples p95 cannot sneak through. For math and code, checkable oracles beat a judge. For policy, faithfulness to retrieved text still wins. See the evaluation FAQ for the machinery.
Can we fine-tune a reasoning model on our data?
Sometimes, on open weights or when the provider offers post-training. Most hosted o-class APIs do not let you train the thinking model itself. Distilling teacher traces into a smaller student is the usual enterprise path. You own those weights and datasets at handover. We will not promise a custom o1; we will say which stack actually fine-tunes.
How does this change a four-week implementation?
It usually does not change the calendar: discovery, environments, shadow-mode, handover. What changes is the routing table and the eval that decides who gets the slow model. A research project to train a new reasoning architecture is not a four-week job. We will say if you are asking for that.
Who pays for the extra tokens?
You, to the model provider, at your contract rates. We add no token markup. Thinking tokens are easy to miss in a prototype and ugly in month two. We put p50/p95 token counts on the dashboard next to quality. If the delta on the golden set is not worth the bill, we leave the reasoning model off that step.
Reasoning in production
Production is budgets, pins, and logs that do not turn a thought stream into a data leak.
Can we use reasoning models if we have a latency SLO?
Yes, off the hot path or with a hard timeout and a fast fallback. Voice agents almost never want a long think on the conversational turn. Back-office document work can wait five seconds. Set the budget in config, fail closed to a cheaper model, and measure both quality and p95. An SLO that assumes a chat model will not survive an unbounded thinking slider.
Can reasoning models call tools?
Many can, and they should: tools and retrieval are still how you ground them. A long think that never looks up the order is an expensive guess. Tool schemas, permissions, and confirmations do not go away because the model is “smarter.” We keep the same allowlists as any other agent.
What happens when the provider changes the reasoning model?
Pinned versions where the API allows it, evals on a live sample, and a page if quality, cost, or latency move. Thinking models change behaviour more than people expect because the hidden budget is part of the product. Silent aliases are how your bill and your SLO move together. Routing stays in your account.
Are open-source reasoning models good enough?
For some code, math, and private-cloud constraints, yes. For the hardest hosted models, often not yet — measure, do not assume. Self-hosting helps residency and cost at high volume. It does not remove evals. We will recommend open weights when the golden set and the GPU budget say so, not as a slogan.
Should we log the thinking traces?
Log what you need to reconstruct a failure: inputs, tools, final answer, model version, token counts. Raw thoughts are optional, sensitive, and sometimes forbidden to display. Default to redaction and a short retention. “Keep all thoughts forever for research” is how you collect PII you cannot erase. Your policy wins; we implement it.
What does handover look like for a routing setup?
The router config, pinned model IDs, eval suite, cost/latency dashboards, runbook, and 30 days of on-call. You own it. Changing the thinking budget after we leave is a config change your engineers can make. We do not keep a hidden prompt that only works on our keys.
Where thinking pays — and where it does not
Buy extra compute where the answer is checkable or the plan is expensive to get wrong. Skip it where retrieval already decides.
Are reasoning models for math and code, or for customer support?
They pay off most on math, code, and planning. Support is usually retrieval, tone, and tools. A reasoning model that invents a refund policy more slowly is not an upgrade. Use them in support only for the rare multi-step exception your eval shows the fast model failing. Head-of-queue intents should stay cheap.
Can we use them for legal or medical reasoning?
Not as an unsupervised authority. A longer chain of thought is still not a clinician or a solicitor. Admin workflows, with retrieval and a human on high-stakes output, are in scope. Diagnosis and formal legal advice are not. Those programmes run longer than four weeks and still need missed-escalation targets. Fluency is the risk.
Do reasoning models help multi-agent planning?
They can help a planner decompose a task, then cheap workers execute tools. That is a good split when the plan is the hard part. It is a bad split when every worker also “thinks” for seconds. See multi-agent orchestration for the architecture. Cap the planner’s budget and eval plan quality separately from worker success.
Can we distill reasoning into a smaller model?
Often that is the point: generate traces with a teacher, filter them, train a student you can host. Collapse and license rules from synthetic data still apply. Distillation is worth it at high volume or when residency forbids the teacher in production. You own the student weights. Eval on real cases, not on more teacher traces.
When should we not use a reasoning model?
When the fast model already clears the eval, when you are on a voice latency budget, when the task is lookup-and-cite, or when you cannot afford unbounded tokens. Also when you only wanted a nicer demo. We will leave the SKU off the graph and say why. Extra thinking is a lever, not a personality upgrade.
Reasoning models spend test-time compute — extra tokens, extra seconds — on hard items: math, code, planning, tool sequencing. They still hallucinate, and the chain of thought is not a legal proof. Production agents should route easy turns to a fast model and pin reasoning models to steps that pay for the wait. ReinforcedX is model-agnostic: we wire routing, evals, and fallbacks in your VPC. You pay the provider; we add no token markup. Four weeks includes the router, not a research programme on new architectures.
Keep reading
Ready to refine
your workflow?
Share your current process. We’ll help you identify what can be automated — and where efficiency can be reclaimed.