Why the Same Question Gets Different Answers

What's for lunch?

Ask someone what they want for lunch and a creature of habit might say the same sandwich every single time, the safest, most predictable answer. Someone more adventurous might name something different depending on their mood, occasionally picking something unexpected. Neither person is wrong; they're just choosing differently between "go with the usual" and "take a chance on something else."

A language model makes a strikingly similar choice at every single token it produces, and a setting called temperature controls how it leans.

It's not always picking the single best guess

At each step, the model doesn't compute one single "correct" next token. It computes a whole set of candidates, each with a likelihood attached, essentially a ranked list of "sandwich," "salad," "soup," and so on, each with a percentage chance attached. Temperature decides how that list gets turned into an actual choice.

This is why asking the same question twice can produce two different, both reasonable, answers: the model isn't retrieving a single fixed response from memory. It's rolling a weighted die over its candidate list every single time, and unless that die is set to always take the safest option, it won't always land the same way.

Why not just always pick the safest option?

For some tasks, that would actually be ideal, and systems built for consistency often do turn temperature down close to zero for exactly that reason. But always taking the safest option has a cost: it tends to produce flatter, more repetitive, more generic-sounding text. A little willingness to take a less obvious path, rather than always defaulting to the most statistically expected word, is part of what makes generated writing feel varied rather than robotic.