What is a context window in an LLM?
A context window is the maximum amount of text, measured in tokens, that a large language model can take into account at one time. Everything counts against it: the system prompt, the conversation so far, any pasted documents or tool results, and the answer the model is writing.
A context window is the model's working memory for a single request, and it is measured in tokens, not words. If something is not inside the window when the model generates its answer, the model cannot see it. There is no background memory it can dip into.
What counts toward the context window?
Every token the model reads or writes in one request counts toward the context window. People often think only of their own message, but the window fills from several places at once:
| What goes in | Typical size | Easy to forget? |
|---|---|---|
| System prompt and instructions | a few hundred to several thousand tokens | Yes, it is sent on every turn |
| Earlier turns of the chat | grows with every message | Yes, the whole history is re-sent |
| Pasted documents and files | can be tens of thousands of tokens | No |
| Tool definitions and tool results | hundreds per tool, results can be huge | Very |
| The model's own reply | up to the output limit you set | Yes |
Two consequences follow. First, a chat app does not "remember" your earlier messages; it re-sends them each time, and they take up room on every turn. Second, the reply needs space too. If your input fills the window, the model has nothing left to answer with.
How big is a context window?
Context windows have grown by roughly three orders of magnitude in five years. GPT-3 in 2020 could read 2,048 tokens, about three pages of text. GPT-4 launched in 2023 with 8,192 and 32,768 token versions. Anthropic's Claude 2 offered 100,000 tokens the same year, and Google's Gemini 1.5 Pro reached 1 million tokens in 2024. Current frontier models commonly accept hundreds of thousands of tokens, and some offer a million or more.
A useful rule of thumb for English text is that one token is about three quarters of a word, so 100,000 tokens is roughly 75,000 words, the length of a novel. Code, non-English text and data such as JSON use more tokens per word. Check the provider's model page for the exact limit, because it changes with every release and there is often a separate, smaller limit on output tokens.
Why do models have a context limit at all?
The limit comes from how transformers work. In the attention step, each token is compared with other tokens to decide what matters, and in the standard design that comparison grows with the square of the sequence length. Doubling the input roughly quadruples that part of the work. Memory is the other constraint: during generation the model keeps a cache of intermediate values for every token already in the window, and that cache grows with every token.
Longer windows are possible through engineering (more efficient attention kernels, better position encodings, training on long documents), but they still cost more per request. Providers usually price by the token, so a request that fills a large window is slower and more expensive than a short one.
Does the model use everything in the window equally well?
No, and this is the part most people miss. A 2023 study from Stanford and collaborators, titled "Lost in the Middle", tested models on questions whose answer sat at different positions in a long input. Accuracy was highest when the relevant passage was at the start or the end, and dropped noticeably when it was buried in the middle. Newer models have improved, but the practical lesson holds: fitting in the window is not the same as being used well.
- Do: Put the task and the most important instructions at the start or the end of a long prompt
- Do: Remove documents and tool output the model does not need for this step
- Do: Ask the model to quote the passage it relied on, so you can check it found the right one
- Avoid: Paste an entire codebase or report "just in case"
- Avoid: Assume a long chat still honours an instruction from forty turns ago
- Avoid: Treat a bigger window as a replacement for choosing what to include
What happens when a conversation exceeds the context window?
When the total goes over the limit, something has to give, and what happens depends on the application, not the model. An API call that is too long is usually rejected with an error. Chat products handle it for you, typically by dropping the oldest turns or replacing them with a summary. Either way, anything that falls out is genuinely gone for the model. That is why a long chat can seem to "forget" a detail you gave early on.
Agent frameworks face the same problem at a larger scale, since every tool call adds its result to the window. Common strategies are summarising old steps, storing notes outside the window and reading them back when needed, and starting a fresh sub-task with only the relevant facts. Managing this deliberately is what people now call context engineering.
How is a context window different from memory or RAG?
A context window is what the model sees during one request. Memory features and retrieval are systems built around the model that decide what to put into that window. Retrieval-augmented generation searches a document store and inserts only the most relevant passages, which is usually cheaper and more accurate than pasting everything. Chat "memory" features save facts between sessions and add them to future prompts. Both still have to fit inside the window to have any effect.
- Context window
- The maximum number of tokens the model can process in one request, input and output together
- Output limit
- A separate cap on how many tokens the model may generate in its reply
- Token
- The unit models read and write, often a word piece; see tokens
- KV cache
- Stored attention values for tokens already processed, the main memory cost of long contexts
- Truncation
- Cutting old content so the rest fits, done by the application rather than the model
Frequently asked questions
Is the context window the same as the model's memory?
No. The context window only covers the current request. A model has no memory between requests unless the application re-sends earlier content or saved notes as part of the next prompt.
Do output tokens count toward the context window?
Yes. The window covers input and output together, and most APIs also set a separate, smaller maximum for output tokens. A prompt that nearly fills the window leaves little room for the answer.
How many words fit in a 128K context window?
Roughly 96,000 words of ordinary English prose, using the rule of thumb that one token is about three quarters of a word. Code, tables, JSON and non-English text use more tokens per word, so fewer words fit.
Is a bigger context window always better?
Not always. Larger inputs cost more, respond more slowly and can bury the important part in the middle, where models use information less reliably. Sending a smaller, well-chosen context often gives better answers than filling the window.
Why does ChatGPT or Claude forget things in a long chat?
Once the conversation grows past the context window, the app drops or summarises older turns so the rest fits. Details from those turns are no longer visible to the model, so it cannot use them.
Last checked for accuracy on . Written by the solidcoder team.