Reasoning models: when the extra latency and cost pay off

GuideLast reviewed Oct 16, 2026By Himanshu Ramchandani

Short answer

A reasoning model generates a chain of thinking tokens before its final answer, working through the problem step by step, and it has been trained specifically to do that well. That makes it much better at problems with many dependent steps, like maths, complex code, planning and careful analysis, and slower and more expensive for everything, because the thinking tokens take time and are billed as output. Use reasoning for the hard steps of your system, use fast standard models for lookups, extraction, classification and formatting, and decide with your own evaluation which tasks are which.

From "think step by step" to reasoning models

In 2022, researchers found that asking a model to show its working before giving the answer, called chain-of-thought prompting, made it much better at maths and logic problems. Even adding the phrase "Let's think step by step" to a prompt improved results on many reasoning tasks.

The reason is mechanical. A model generates one token at a time, and each token is computed with the same fixed amount of work. If the model has to give the answer to a hard question as its first token, it gets very little computation to find it. If it writes out the steps first, every step it writes becomes input for the next one, so it effectively gets more computation, and intermediate results it can build on.

Reasoning models take this further. Instead of relying on a prompt to make the model show its work, they are trained, mostly with reinforcement learning on problems with checkable answers, to produce long, useful chains of reasoning on their own: trying an approach, checking it, noticing a mistake and backtracking. OpenAI's o1, released in 2024, brought this into wide use, DeepSeek-R1 in January 2025 showed an open-weight reasoning model and published how it was trained, and most major model families now offer a reasoning mode.

The idea behind them is sometimes called test-time compute: instead of only making models bigger during training, you let them spend more computation when they answer.

How they work from your side

When you call a reasoning model, three things are different.

The model thinks before it answers. Depending on the provider, you may see the full thinking, a summary of it, or nothing at all, but it happens either way.

You pay for the thinking. Thinking tokens are billed as output tokens, the expensive kind, and a model can generate thousands of them before writing a short answer. The usage section of the API response usually reports reasoning tokens separately, so log them.

You can often control how much it thinks. Many APIs offer a setting like a reasoning effort level (low, medium, high) or a thinking budget in tokens. More thinking usually means better answers on hard problems, and always means more time and cost.

When reasoning pays off

Reasoning models are worth the extra time and cost when the task has several dependent steps and getting one step wrong ruins the result.

Maths and quantitative analysis: calculations with many steps, financial models, statistics.
Complex code: debugging a problem across several files, designing an algorithm, refactoring with constraints, writing code that has to be right the first time.
Planning: breaking a goal into steps, ordering tasks with dependencies, an agent deciding how to approach a multi-step job.
Careful analysis of documents against rules: checking a contract against a policy, comparing several sources that disagree.
Problems where you would want a person to stop and think before answering.

When it doesn't

For many tasks, a reasoning model is slower, more expensive and no better, and sometimes worse, because it overthinks a simple question.

Lookups and simple questions: "What is the capital of India?" doesn't need a plan.
Extraction and classification: pulling fields from an invoice or labelling a ticket.
Formatting and rewriting: converting text to JSON, changing tone, summarising a short text.
Chat where speed matters: users wait several seconds or more while the model thinks.
High-volume pipelines, where the extra tokens multiply into a large bill.

Use both: route by difficulty

The best systems rarely use one kind of model for everything. A common pattern is to send most requests to a fast standard model, and only the hard ones to a reasoning model.

You can decide which is which with simple rules, like the type of task, with a small classifier model, or by letting the fast model try first and escalating when its answer fails validation. In an agent, a reasoning model can make the plan, and faster models can carry out the individual steps.

Prompting reasoning models

Reasoning models respond to prompts differently from standard models.

State the goal, the constraints and what a good result looks like, and let the model work out the steps. Long step-by-step instructions often help less than with standard models.
You usually don't need to ask it to think step by step. It already does.
Give it the full context it needs in one go, rather than asking it to guess.
Ask for the final answer in a clear format, separate from the reasoning, so your code can read it.
Check the provider's guidance, because each model family has its own recommendations, and some don't support settings like temperature.

Reasoning is not the same as being right

A reasoning model can still make mistakes, and it can make them with a long, convincing explanation.

It can still hallucinate facts, because thinking more doesn't give it knowledge it doesn't have. It is still bad at exact arithmetic on long numbers compared with a calculator, so give it a code tool for exact calculations. And the reasoning it shows doesn't always reflect how it really reached the answer: research from Anthropic in 2025 found that reasoning models often didn't mention hints in their visible reasoning even when those hints had changed their answer. So treat the visible reasoning as a useful explanation, not as proof.

How to decide with evidence

Don't switch to a reasoning model because it is new, or switch everything because it helped once. Measure it on your task.

Take your evaluation set and tag each example by type or difficulty.
Run it with your current model and with a reasoning model at one or two effort levels.
Compare accuracy, latency and cost per request for each type of example.
Use reasoning only for the types where the accuracy gain is worth the extra time and cost.

You will often find that reasoning helps a lot on a small share of hard requests and not at all on the rest, which is exactly the case for routing.

Running a reasoning model locally

Open-weight reasoning models can run on your own machine. Distilled versions of DeepSeek-R1, smaller models trained to imitate its reasoning, are available in sizes from about 1.5 billion parameters, and run through tools like Ollama.

ollama pull deepseek-r1:8b
ollama run deepseek-r1:8b

You will see the model write its thinking before the answer, which is a good way to watch what reasoning looks like and how many tokens it takes. Small distilled models reason much less reliably than the large ones, so use them to learn and for narrow tasks you have tested.

From the floor

When we build agents in our cohorts, I teach reasoning first as a prompting pattern: ReAct, where the agent reasons, acts and observes in a loop, chain of thought, where it works through intermediate steps, and tree of thoughts, where it looks ahead at different options before choosing. Reasoning models build the first part of that into the model itself, but the loop around it, calling tools, checking results and deciding the next step, is still something you design.

In the session on running LLMs locally, the model we installed was an 8 billion parameter DeepSeek-R1 model, about 4.9 GB, pulled and run with Ollama and given a browser interface with Open WebUI.

Next step

Read multimodal models.

Sources

← PreviousLLM pricing math: cost per request, per user and per month Next in section →Multimodal models: vision, audio and documents