What is a transformer, and how does attention work?

A transformer is the neural network architecture behind modern large language models. Its key idea, self-attention, lets every token in the input look at every other token and decide which ones matter for understanding it. Introduced by Google researchers in 2017, it replaced older sequential designs because it trains efficiently in parallel.

3 min read·Checked September 25, 2026·Also called transformer architecture, self-attention, attention mechanism

Read the free chapter →How AI Actually Thinks guide

Attention is how a model decides which earlier words matter for the word it is working on. In "The trophy did not fit in the suitcase because it was too big", attention is what lets the model connect "it" to "trophy" rather than "suitcase".

What problem did the transformer solve?

Before 2017, most language models were recurrent neural networks, which read text one word at a time and passed a summary along to the next step. That design had two problems: information from far back in a sentence faded, and training could not be parallelised well because each step waited for the previous one. The 2017 paper "Attention Is All You Need", by eight researchers mostly at Google, removed the recurrence entirely and built the model from attention and simple feed-forward layers. Every position could now be processed at once on a GPU, which is what made training on internet-scale text practical.

How does self-attention work?

Self-attention works by letting each token ask a question of every other token and take a weighted mix of the answers. For each token, the model computes three vectors from its embedding:

Query: What this token is looking for, for example "which noun do I refer to?"
Key: What each token offers to be matched against
Value: The information a token passes on if it is attended to

The model compares one token's query with every token's key to get a score, turns the scores into weights that sum to one (using a softmax), and takes the weighted sum of the values. The result is a new vector for that token that blends in information from the tokens it found relevant. In a language model that generates text, each token may only attend to tokens before it, so the model cannot peek at the answer it is about to write.

Transformers run many attention calculations side by side, called attention heads, so different heads can track different relationships, such as grammar, coreference or matching brackets in code. A full model stacks dozens of these layers, each followed by a feed-forward network, and the vectors get richer at every layer.

What are the parts of a transformer?

1
Tokenize and embed
The text is split into tokens and each token ID becomes a vector. Position information is added so the model knows word order.
2
Attention layers
Each layer lets every token gather information from the relevant tokens around it.
3
Feed-forward layers
After attention, each token's vector passes through a small network that transforms it further; much of the model's stored knowledge is thought to live here.
4
Repeat
Frontier models stack many such layers; the original paper used six in each half.
5
Predict
The final vector for the last position is turned into a probability for every token in the vocabulary, and the next token is chosen.

The original transformer had an encoder, which read the input, and a decoder, which wrote the output, because it was built for translation. GPT-style LLMs use only the decoder half. BERT-style models, often used for classification and embeddings, use only the encoder half.

Why does attention make long contexts expensive?

Standard attention compares every token with every other token, so the work grows with the square of the input length. Double the input and that part of the computation roughly quadruples. The model also keeps a cache of keys and values for every token already processed, which grows linearly and consumes GPU memory. These costs are why context windows have limits and why long prompts cost more.

A lot of engineering goes into softening this. FlashAttention, published in 2022, reorganised the computation to make far better use of GPU memory without changing the result. Other techniques share keys and values across heads or limit how far some layers look back.

Why should an AI engineer care about attention?

Because it explains behaviour you will see every day. Models attend unevenly across long inputs, which is why information buried in the middle of a long prompt is used less reliably than information at the start or end. Clear structure, such as headings, labelled sections and XML-style tags around documents, gives attention something to latch onto. And because every token can influence every later token, one stray instruction inside a document can change the model's behaviour, which is the root of prompt injection.

Frequently asked questions

What does GPT stand for?

GPT stands for Generative Pre-trained Transformer. "Generative" because it produces text, "pre-trained" because it first learns from a large corpus before any task-specific training, and "transformer" for the architecture.

Is every LLM a transformer?

Nearly all widely used LLMs are transformer-based. Alternatives such as state-space models exist and some models mix them with attention layers, but the transformer remains the dominant design.

What is the difference between attention and self-attention?

Attention is the general mechanism of weighting inputs by relevance. Self-attention is the case where a sequence attends to itself, so each token looks at the other tokens in the same input. Transformers are built mainly from self-attention.

What is multi-head attention?

Multi-head attention runs several attention calculations in parallel, each with its own learned weights. Different heads can focus on different kinds of relationships, and their outputs are combined before the next layer.

Why is it called "Attention Is All You Need"?

The title of the 2017 paper made the point that a model built from attention alone, without the recurrent or convolutional layers used before, could match or beat earlier translation systems while training much faster.

Last checked for accuracy on September 25, 2026. Written by the solidcoder team.