How a language model writes

A chatbot seems to write whole answers, but underneath it does one small thing over and over: look at the text so far, give every possible next token a probability, pick one, append it, and repeat. A long answer is a few hundred of these picks.

This lesson takes that loop apart. First, what a token is: not a word or a letter, but a piece learned from data. Then how a probability for every token becomes one choice, and the knobs (temperature, top-k, top-p) that decide how bold the choice is. Last, the trick that keeps the loop fast.

token

The unit a model reads and writes: a whole common word, a piece of a rarer one, or a single character. Each has a number in the vocabulary.

byte-pair encoding (BPE)

A way to learn the vocabulary: start from characters and keep gluing together the pair of neighbours that occurs most often.

logit

The raw score the model's last layer gives one token. Higher means likelier, but logits are not probabilities yet.

softmax

Turns logits into probabilities: $p_i = e^{z_i / T} / \sum_j e^{z_j / T}$, all positive and summing to 1.

temperature T

Divides the logits before the softmax. Below 1 sharpens the distribution, above 1 flattens it, and $T = 0$ means always take the top token.

top-k and top-p

Cuts before drawing: keep only the $k$ likeliest tokens, or the fewest whose probabilities add up to $p$.

Every merge makes the text a little shorter, and the merges are kept in order. To tokenize new text, the tokenizer replays them: a word it never saw is still built from pieces it knows, so there is no such thing as an unknown word.

The toy model below counts which word follows which in a short story and uses those counts as logits. A real model gets its logits from a transformer, but everything after that is exactly the same.

At $T = 0$ there is no randomness at all. Predict each word before it appears, and watch what happens when a context comes round a second time.

Switch between the examples, or write your own settings in the input box (for example temperature = 2, top-k = 0, top-p = 1, seed = 4, length = 8, prompt = the dog):

• T = 0.7 against T = 1.5: which one keeps to the story, and which one wanders?
• Top-k 3: after a word the model is sure about, does keeping three words still make sense?
• Top-p 0.9: count how many words survive after "the", then after "sat". Why does the number change?

Each new token needs attention over every token before it. Recomputing every earlier token's keys and values at each step would repeat almost all of the work, so models keep them in a KV cache and compute only the new token's row.

The cache turns the cost of each step from growing with the square of the text into growing with its length, at the price of memory: long conversations are expensive mostly because their caches are large.

• Models read and write tokens: pieces learned by BPE, from whole common words down to single characters.
• At each step the model gives every token a logit; softmax with temperature turns them into probabilities.
• Top-k and top-p cut the unlikely tail, then a random number picks one token. T = 0 always takes the top one, and loops.
• Keys and values of earlier tokens are cached, so each new token costs one row of work instead of the whole prefix.