Sampling: temperature, top-p and top-k explained
At every step, an LLM produces a probability for every token in its vocabulary, and sampling is how one token gets picked. Temperature changes how spread out those probabilities are: low values make the most likely token win almost every time, high values give unlikely tokens more chance. Top-k and top-p cut off the long tail of unlikely tokens before picking. Use low temperature for extraction, classification and code, moderate values for conversation, and higher values only for brainstorming and creative writing, and always check what your provider's defaults are.
From scores to probabilities
When the model has processed your input, its final layer gives a score for every token in the vocabulary. These raw scores are called logits, and they can be any number, positive or negative.
The softmax function turns logits into probabilities. It makes every value positive and makes them add up to 1, while keeping their order: the token with the highest logit gets the highest probability.
For "The sky is", the result might look something like this:
| Token | Probability |
|---|---|
| blue | 0.62 |
| clear | 0.14 |
| dark | 0.09 |
| grey | 0.05 |
| everything else | 0.10 spread over 100000+ tokens |
Now the model has to pick one. That choice is sampling.
Greedy decoding
The simplest choice is to always pick the most likely token. This is called greedy decoding.
It is predictable, but it has a known problem for longer text: always taking the single most likely next word often leads to bland, repetitive output, sometimes looping on the same phrase, because the most likely continuation of a repetitive text is more repetition.
So most applications sample instead, picking a token at random in proportion to its probability. "blue" will be chosen about 62% of the time, "clear" about 14%, and so on. That randomness is why the same prompt can give different answers.
Temperature
Temperature controls how much randomness there is. Before softmax, every logit is divided by the temperature.
- Temperature 1 leaves the probabilities as the model produced them.
- Temperature below 1 makes them sharper: likely tokens become even more likely, and unlikely ones almost disappear. Close to 0, it behaves like greedy decoding.
- Temperature above 1 makes them flatter: unlikely tokens get a much bigger share, so output becomes more varied and, at high values, incoherent.
You can see the effect with a few lines of code.
import numpy as np
def softmax(logits: np.ndarray, temperature: float) -> np.ndarray:
scaled = logits / temperature
exp = np.exp(scaled - scaled.max())
return exp / exp.sum()
tokens = ["blue", "clear", "dark", "grey", "purple"]
logits = np.array([4.0, 2.5, 2.1, 1.5, -1.0])
for t in [0.2, 0.7, 1.0, 1.5]:
probs = softmax(logits, t)
print(t, {tok: round(float(p), 3) for tok, p in zip(tokens, probs)})
At temperature 0.2, "blue" gets almost all the probability. At 1.5, "purple", which almost never appears at low temperatures, becomes about four times more likely than at temperature 1.
People often describe temperature as creativity. That is a useful shorthand, but more precisely it is randomness: higher temperature doesn't make the model more imaginative, it makes it more willing to pick tokens it considered less likely.
Top-k
Top-k sampling keeps only the k most likely tokens and samples among them. With top-k of 40, the model picks only from its 40 best options, and the long tail of very unlikely tokens, which can contain nonsense, is never chosen.
The weakness of top-k is that the right number of options changes from step to step. After "The capital of France is", there is really one good answer, while after "My favourite colour is", there are many, and a fixed k is too loose in the first case or too tight in the second.
Top-p (nucleus sampling)
Top-p solves that by keeping the smallest set of tokens whose probabilities add up to p. With top-p of 0.9, the model keeps the most likely tokens until their total reaches 90%, and samples only among those.
When the model is confident, that set is small, sometimes one or two tokens. When it is uncertain, the set is larger. So top-p adapts to each step in a way top-k doesn't.
def top_p_filter(probs: np.ndarray, p: float) -> np.ndarray:
order = np.argsort(probs)[::-1]
cumulative = np.cumsum(probs[order])
keep = order[: int(np.searchsorted(cumulative, p)) + 1]
filtered = np.zeros_like(probs)
filtered[keep] = probs[keep]
return filtered / filtered.sum()
probs = softmax(logits, 1.0)
print({tok: round(float(x), 3) for tok, x in zip(tokens, top_p_filter(probs, 0.9))})
Some libraries and open models also support min-p, which keeps tokens whose probability is at least a set fraction of the top token's probability, another way of adapting the cut-off to how confident the model is.
How the settings combine
When several settings are used together, they are usually applied in order: temperature reshapes the probabilities, then top-k and top-p cut the list, then one token is sampled from what is left. The exact order and defaults differ between providers and inference engines, so check the documentation.
A common recommendation from providers is to adjust temperature or top-p, not both at once, because their effects overlap and changing both makes it hard to know which one caused a change in quality.
Other generation settings
A few other parameters often sit next to temperature in an API call.
- Max output tokens: the most tokens the model may generate. Set it to protect your cost and latency, but not so low that answers get cut off in the middle.
- Stop sequences: strings that end generation when the model produces them, useful for keeping the model from continuing past the part you need.
- Frequency and presence penalties: reduce the probability of tokens that have already appeared, to discourage repetition. Use them carefully, because they can also stop the model from repeating words it should repeat, like a product name.
- Seed: some APIs accept a seed so that sampling is repeatable, on a best-effort basis. It reduces variation but doesn't guarantee identical output, for reasons the nondeterminism page explains.
Which settings to use for which task
These are starting points, not rules. Test them with your own evaluation set, because the best values differ between models.
| Task | Temperature | Why |
|---|---|---|
| Extraction, classification, structured output | 0 to 0.2 | You want the most likely answer and consistent formatting |
| Code generation | 0 to 0.3 | Small random changes can break code |
| RAG answers from documents | 0.1 to 0.4 | Stay close to the retrieved text |
| Conversation, support chat | 0.4 to 0.8 | Natural, varied wording without drifting |
| Brainstorming, creative writing | 0.8 to 1.2 | Variety matters more than the single best answer |
A useful habit is to generate several answers at a higher temperature when you want options, and use a low temperature when you want one dependable answer.
Reasoning models and sampling settings
Reasoning models, the ones that think step by step before answering, often handle sampling differently. Some don't accept a temperature setting at all, and others recommend specific values and warn that changing them makes results worse. Read the model's documentation before copying settings from another model.
Where to change these settings
Every major API exposes them as parameters in the request. To get a feel for them before writing code, use a playground like Google AI Studio, where you can move the temperature slider and run the same prompt several times to see how the answers change.
From the floor
In the session on nondeterminism, I opened Google AI Studio and showed the temperature setting on the right side of the screen. One builder asked whether temperature is something you set before asking the question, and yes, it is a setting of each request, not something the model decides on its own, and you can change it for every call. Another asked what temperature means, and the simplest answer I have is that it controls the randomness of which word gets picked next.
Related
- Why LLMs give different answers at temperature 0
- How an LLM works
- Logits and softmax
- Evals and reliability