A language model generates one token at a time, and every new token attends to all earlier ones. Their keys and values never change once computed, so the model keeps them in a cache and computes just one new row of K and V per step instead of all of them again.