What are tokens in AI, and how does tokenization work?
A token is the unit of text a language model reads and writes. It is often a whole common word, a piece of a longer word, a punctuation mark or a space. In English, one token averages about four characters, or roughly three quarters of a word, and API usage is priced and limited by token count.
Models never see letters or words; they see a sequence of token IDs. Most of the practical limits you run into as an AI engineer, from context size to cost to odd spelling mistakes, are measured in tokens, so it pays to know what one is.
How does tokenization work?
Tokenization splits text into pieces from a fixed vocabulary and replaces each piece with an integer ID. The model is trained on those IDs, and when it answers, it outputs IDs that the tokenizer turns back into text. Vocabularies usually hold somewhere between tens of thousands and a few hundred thousand entries.
Most modern tokenizers use a method based on byte pair encoding (BPE), introduced for machine translation by Sennrich and colleagues in 2016. BPE starts from single characters or bytes and repeatedly merges the most frequent adjacent pair into a new token. After enough merges, common words like "the" or " house" become single tokens, while rare words are built from several smaller pieces.
| Text | How it might be split | Tokens |
|---|---|---|
| the | the | 1 |
| unbelievable | un · believ · able | 3 |
| tokenization | token · ization | 2 |
| ChatGPT | Chat · G · PT | 3 |
| 2026-09-25 | 202 · 6 · - · 09 · - · 25 | 6 |
The splits above are illustrative. Each model family has its own tokenizer, so the same sentence can be a different number of tokens on different models. Note that a leading space is usually part of the token, so " cat" and "cat" are different tokens.
How many tokens is a word?
In typical English prose, one token is about four characters, so 100 tokens is roughly 75 words. OpenAI's own guidance uses this rule of thumb. The ratio gets worse for other content:
| Content | Tokens compared with English prose | Why |
|---|---|---|
| Plain English prose | Baseline, about 0.75 words per token | Common words are single tokens |
| Source code | More per line | Symbols, indentation and long identifiers split into pieces |
| JSON and tables | Noticeably more | Braces, quotes and repeated keys each cost tokens |
| Many non-English languages | Often two to three times as many | Scripts less common in training data get split finely |
| Numbers, dates and IDs | More than they look | Digits are broken into short, uneven chunks |
For exact numbers, use the provider's tokenizer or token-counting endpoint rather than an estimate. OpenAI publishes its tokenizer as the open-source tiktoken library, and Anthropic offers a token-counting API.
Why do tokens matter for AI engineers?
Tokens matter because nearly every limit and cost in an LLM system is counted in them. The context window is a token limit. API prices are quoted per million input and output tokens, with output tokens usually costing several times more than input. Response time grows with the number of output tokens, because they are generated one after another. Rate limits are often set in tokens per minute.
This makes token counting a basic engineering habit. A system prompt that grows from 500 to 5,000 tokens is sent on every request, so it multiplies your input cost by up to ten. See LLM cost and latency for ways to keep this under control.
Why can't LLMs count the letters in a word?
Because the model never sees the letters. Asked how many times "r" appears in "strawberry", a model sees two or three tokens, not ten characters, so it has to rely on what it learned about the spelling of those tokens during training. The same effect makes models unreliable at reversing strings, counting characters and some arithmetic, where numbers are split into uneven chunks. Newer models handle many of these cases better, and giving the model a code tool removes the problem entirely.
What is the difference between tokens and embeddings?
A token is a discrete ID for a piece of text. An embedding is a list of numbers that represents meaning. Inside the model, each token ID is first looked up in a table that turns it into an embedding vector, and all later computation works on those vectors. So tokenization is the step before embeddings: text becomes token IDs, and token IDs become vectors.
- Token
- A piece of text from the model's vocabulary, identified by an integer ID
- Tokenizer
- The component that converts text to token IDs and back
- Vocabulary
- The fixed set of tokens a model knows, often 50,000 to 250,000 entries
- BPE
- Byte pair encoding, the merge-based method most LLM tokenizers are built on
- Input and output tokens
- Tokens you send and tokens the model generates, usually priced differently
Frequently asked questions
How many tokens are in 1,000 words?
About 1,300 to 1,400 tokens for ordinary English prose, using the rule of thumb of roughly 0.75 words per token. Code, JSON and many non-English languages use more.
Are tokens the same across ChatGPT, Claude and Gemini?
No. Each model family uses its own tokenizer and vocabulary, so the same text produces different token counts. Compare prices using each provider's own token counts, not a single estimate.
Do spaces and punctuation count as tokens?
Yes. Punctuation marks are usually their own tokens, and spaces are typically attached to the start of the following word. Extra whitespace and formatting therefore add to your token count.
Why are output tokens more expensive than input tokens?
Input tokens can be processed in parallel in a single pass, while output tokens are generated one at a time, each needing its own pass through the model. That sequential work costs more compute per token.
How can I reduce the number of tokens I use?
Shorten system prompts, send only the documents and history that the current step needs, ask for concise outputs, and use prompt caching for large repeated prefixes. Measure with the provider's token counter before and after each change.
Last checked for accuracy on . Written by the solidcoder team.