Two small numbers in every LLM app — temperature and top-p — control how creative or predictable the model's answers get. This guide explains what each setting actually changes inside the model, and how to set them for different tasks, from factual writing to brainstorming.
Temperature: How Random the Model Gets
Under the hood, a language model does not pick the single "best" next word. It scores thousands of candidate words, converts those scores into probabilities, and samples from them. Temperature controls how sharply those probabilities are skewed.
At a temperature of zero, the model becomes nearly deterministic: it almost always picks the highest-probability word. That is what you want for factual tasks — summaries, code, data extraction — where creativity is a liability. Raise the temperature toward 1, and the model starts sampling from lower-probability words, producing more varied and surprising output. Push it much higher and the text degenerates into incoherence.
The practical rule: lower temperature for anything where correctness matters, higher temperature when you want diversity — brainstorming, creative writing, generating many options. Most chat apps default to around 0.7, a compromise between the two. Also note that even at temperature zero, large models are not perfectly deterministic across different hardware and software versions; do not rely on exact reproducibility for critical systems.
Top-p: A Smarter Way to Limit Choices
Top-p (nucleus sampling) is a second control that works alongside temperature. Instead of considering every possible next word, the model keeps only the smallest set of words whose combined probability reaches p. With top-p set to 0.9, the model samples from words covering the top 90 percent of probability mass and ignores the long tail entirely.
The advantage over a fixed cutoff: top-p adapts to the model's confidence. When the model is sure about the next word (a factual answer), the candidate set shrinks to a handful of words. When it is uncertain (an open-ended question), the set widens automatically. That makes it a gentler way to reduce weird outputs than simply lowering temperature.
Most practitioners adjust one setting, not both. A common setup: keep temperature around 0.7–1.0 and lower top-p to 0.9 to trim the tail, or set top-p to 1 and vary temperature alone. Changing both aggressively can interact in confusing ways.
Why it matters
Most people never touch these settings and then complain the model is either boring or unreliable. They are the closest thing an LLM has to a behavior dial — and they cost nothing to change. The next time a model gives you a dull brainstorm or a confidently wrong summary, do not rewrite your prompt first. Check the temperature. Nine times out of ten, the prompt was fine; the sampling was wrong.
A VCTRIA evergreen guide.