Attention: every token looks at every other

In "the cat ate because it was hungry", you know at once that it is the cat. A model reading the sentence has to work that out too: the word it means nothing on its own until it borrows meaning from another word.

Attention is how a transformer does the borrowing. Every token asks a question, every token offers an answer, and each token blends in information from the tokens whose answers fit its question best. Every large language model is built from stacks of this one operation.

query q

What a token is looking for. "it" looks for a noun.

key k

What a token offers to others. "cat" advertises itself as a noun.

value v

The information a token hands over to whoever attends to it.

score

The dot product $q \cdot k$, divided by $\sqrt{d}$ (the number of dimensions): large when the query and key point the same way.

softmax

Turns scores into weights between 0 and 1 that add up to 1: $w_j = e^{s_j} / \sum_i e^{s_i}$.

mask

In a decoder like GPT, a token may not look at later tokens: their scores become $-\infty$, so their weights are exactly 0.

Here every vector has just two numbers, so you can see them as arrows; real models use 64 or 128 per head, and learn them from data instead of having them set by hand. The recipe is the same:

Doing this for every token at once gives a matrix of weights, one row per query: $\operatorname{softmax}(QK^\top/\sqrt{d})\,V$. With $n$ tokens that is $n^2$ scores, which is why long contexts are expensive.

A decoder writes one token at a time, and each new token attends to everything before it. The keys and values of the earlier tokens don't change, so the model keeps them in a KV cache and computes only one new row per step.

Drag the query arrow q, or type a query into the box:

• Make "it" attend to "ate". Which direction does q have to point?
• Set q to (0, 0). Every score is 0, so every token gets the same weight: the output is the plain average of the values.
• Make q long. The scores spread apart and softmax picks one winner almost outright. That is why scores are divided by $\sqrt{d}$.

• Each token has a query (what it wants), a key (what it offers) and a value (what it hands over).
• Scores are $q \cdot k / \sqrt{d}$; softmax turns them into weights that add up to 1; the output is $\sum_j w_j v_j$.
• A mask hides later tokens from a decoder, so it can learn to predict the next token.
• While generating, a KV cache keeps the keys and values of earlier tokens: one new row per step, at the cost of memory.