Paying Attention to What Matters
Not every earlier word matters equally for the next guess. Attention is how a model decides which ones to weigh most heavily.
A sentence that needs a re-read
Read this sentence once: "The trophy didn't fit in the suitcase because it was too big." Now answer quickly: what was too big, the trophy or the suitcase?
Most people say the trophy. But notice what you had to do to answer that: you couldn't just look at the word "it" on its own, since "it" doesn't tell you anything by itself. You had to reach back across the sentence and connect "it" to "trophy," while also considering "suitcase" and ruling it out because a suitcase being big wouldn't explain why the trophy didn't fit inside it.
That reaching back and weighing which earlier words actually matter for understanding the current one is, in essence, what a mechanism called attention does inside a language model.
Not every earlier word counts the same
Without something like attention, a model predicting the next token after "it was too" would have to treat every earlier word in the sentence as equally relevant, "the," "didn't," "trophy," "suitcase," all given the same weight. That's clearly not how the sentence actually works. "Trophy" matters enormously for predicting what comes after "it." "Didn't" matters much less.
Attention lets the model, at every single step of guessing, look back across everything it has seen so far and assign each earlier token a different amount of relevance to the current guess. For this particular prediction, "trophy" might get weighed heavily, "suitcase" moderately, and "the" barely at all. For a different prediction, later in the same sentence, the weighting might look completely different.
Why this is the part that made modern AI possible
Earlier approaches to this kind of prediction mostly processed text in strict order, one word affecting the next, with older information fading out the further back it was. That made long-range connections, like "it" pointing back to "trophy" several words earlier, hard to hold onto reliably.
Attention removed that limitation almost entirely. A word at the very start of a paragraph can still directly influence a prediction near the end, if it's actually relevant, because the model isn't relying on information trickling forward step by step. It's looking back across everything at once and deciding, fresh, what matters.
"The city council refused the protesters a permit because they feared violence." Who feared violence, the council or the protesters? Try a version where you swap "feared" for "advocated," and notice how your answer flips even though almost nothing else in the sentence changed. That sensitivity to which word "they" should connect back to is exactly the kind of judgment attention is doing, constantly, while a model generates text.