What is temperature in an LLM?
Temperature is a setting that controls how much randomness a language model uses when choosing each next token. A low temperature makes it pick the most likely tokens, giving focused and repeatable answers. A higher temperature spreads the choice across less likely tokens, giving more varied and creative, but less predictable, output.
The model does not choose words; it produces probabilities, and a separate sampling step chooses. Temperature is the main dial on that sampling step, and it is why the same prompt can give a different answer every time.
How does temperature work?
At each step, a large language model outputs a score for every token in its vocabulary. These scores are divided by the temperature and turned into probabilities. Dividing by a small number exaggerates the gaps, so the top token gets almost all the probability. Dividing by a large number shrinks the gaps, so less likely tokens get a real chance of being picked.
Suppose the model is completing "The capital of France is" and its probabilities are 90% " Paris", 5% " a", 2% " the" and small amounts for thousands of others.
| Temperature | Effect on this example | Typical use |
|---|---|---|
| 0 or near 0 | " Paris" almost every time | Extraction, classification, code, factual answers |
| 0.5 to 0.8 | Usually " Paris", occasionally a different phrasing | General chat and writing |
| 1.0 | The model's raw probabilities, unchanged | Default on many APIs |
| Above 1 | Unlikely tokens picked noticeably often | Brainstorming, varied drafts; quality drops fast |
Temperature does not add knowledge or creativity; it only changes how adventurously the model samples from what it already predicts. High values make hallucinations and incoherent text more likely, because low-probability tokens are often low probability for good reason.
What are top-p and top-k?
Top-p and top-k are two other ways of limiting which tokens can be sampled, and they are often used alongside temperature.
- Top-k
- Keep only the k most likely tokens, then sample among them; top-k of 1 means always pick the top token
- Top-p (nucleus sampling)
- Keep the smallest set of tokens whose probabilities add up to p, such as 0.9, then sample among them
- Greedy decoding
- Always take the single most likely token, equivalent to a temperature near zero
- Seed
- A value some APIs accept to make sampling repeatable, on a best-effort basis
Nucleus sampling comes from a 2019 paper by Holtzman and colleagues, "The Curious Case of Neural Text Degeneration", which showed that always taking the most likely word produced dull, repetitive text, while sampling from the full distribution produced nonsense. Cutting off the long tail of unlikely tokens gave more natural results. Most providers recommend adjusting temperature or top-p, not both at once.
Why does the same prompt give different answers?
Because sampling is random by design. At any temperature above zero, two runs can diverge at the first token where more than one option has real probability, and once one token differs, everything after it can differ too. For chat and writing this is a feature: you can regenerate to get another version.
Is temperature 0 fully deterministic?
Not always. Setting temperature to 0 makes the model pick the top token at each step, but output can still vary slightly between runs. Tiny floating-point differences in how a server batches and computes requests can change which token comes out on top when two are nearly tied. Researchers at Thinking Machines Lab showed in 2025 that batching is a major source of this and that it can be removed, at some cost in speed. Some providers also offer a seed parameter for more repeatable output. Treat temperature 0 as "highly consistent", not "guaranteed identical".
Some newer reasoning models do not let you set temperature at all, or ignore it, because their training assumes a fixed sampling setup.
What temperature should I use?
Pick the lowest temperature that still gives acceptable variety, and measure it rather than guessing.
- Do: Use a low temperature for extraction, classification, structured output and tool calls
- Do: Use a moderate temperature for conversation and drafting
- Do: Run each eval case several times at your production temperature to see real variance
- Avoid: Raise temperature to fix boring answers; better instructions and examples work better
- Avoid: Compare two prompts using one run each at a high temperature; the difference may be noise
- Avoid: Change temperature and top-p together without a reason
Frequently asked questions
What is a good temperature for ChatGPT or Claude?
For factual and technical tasks, 0 to 0.3 is a common choice. For conversation and writing, the provider default (often around 1) or a slightly lower value works well. The best setting depends on the task, so test it.
Does a higher temperature make the model more creative?
It makes output more varied, not smarter. The model samples less likely words more often, which can produce fresher phrasing but also more errors and less coherent text.
What is the difference between temperature and top-p?
Temperature reshapes the whole probability distribution, making it sharper or flatter. Top-p cuts off the unlikely tail, keeping only the most probable tokens that together reach a set probability. Both control randomness in different ways.
Why do I get different answers at temperature 0?
Server-side factors such as request batching and floating-point arithmetic can change which token wins when two are nearly tied, so outputs can still differ slightly between runs even at temperature 0.
Last checked for accuracy on . Written by the solidcoder team.