What is chunking in RAG, and which strategy should you use?

Chunking is splitting documents into smaller passages before embedding and indexing them for retrieval. Each chunk should be small enough to be specific and cheap to include in a prompt, yet complete enough to make sense on its own. How you chunk often matters more to RAG quality than which model or vector database you use.

3 min read·Checked September 25, 2026·Also called document chunking, text splitting, chunking strategy

Read the free chapter →How AI Actually Thinks guide

A retriever can only return the chunks you created. If the answer to a question was split across two chunks, or buried in a chunk about something else, no amount of tuning later will find it cleanly.

Why does chunking matter?

Chunking decides the units that search can find and the model can read. Chunks that are too large blend several topics, so their embeddings become vague and they waste context window space. Chunks that are too small lose the surrounding context: a chunk that says "The limit is 500 per day" is useless if it does not say which limit, for which plan.

What are the main chunking strategies?

Strategy	How it splits	Strengths	Weaknesses
Fixed-size	Every N tokens, often with overlap	Simple, predictable	Cuts through sentences, tables and sections
Recursive	By paragraphs, then sentences, then words, to stay under a size limit	Respects natural boundaries; a sensible default	Still ignores document structure
Structure-aware	By headings, sections, list items, table rows or code functions	Chunks match how the document is organised	Needs parsing per format
Semantic	Starts a new chunk where the topic shifts, detected with embeddings	Topic-coherent chunks	Slower, and less predictable sizes
Contextual	Any of the above, plus a short generated summary of where the chunk sits in the document	Chunks make sense on their own	Extra LLM cost at indexing time

How big should chunks be?

There is no universal best size; a few hundred tokens per chunk, with some overlap, is a common starting point for prose, and you should tune from there on your own questions. Short, factual lookups tend to favour smaller chunks; questions that need explanation favour larger ones. Overlap, such as repeating the last 10 to 20% of one chunk at the start of the next, reduces the chance that a key sentence is cut in half, at the cost of some duplication.

1
Start simple
Recursive splitting at a few hundred tokens with modest overlap.
2
Build a test set
30 to 100 real questions, each with the passage that answers it.
3
Measure retrieval
Check how often the right passage appears in the top results.
4
Look at misses
Read the failures: split answers, missing context, noisy chunks.
5
Change one thing
Size, overlap, structure-aware splitting or added context, then measure again.

What is contextual chunking?

Contextual chunking adds a short explanation of each chunk's place in its document before embedding it. Anthropic's 2024 contextual retrieval write-up gives an example: a chunk saying "The company's revenue grew by 3% over the previous quarter" is rewritten with a prefix naming the company and the filing period. In Anthropic's tests, this reduced failed retrievals by 35% on its own, 49% combined with keyword search, and 67% with reranking added. The prefixes are generated by an LLM once, at indexing time, and prompt caching keeps that affordable because the whole document is reused for each of its chunks.

A cheaper version is to always prepend the document title and section heading to each chunk.

What should you store with each chunk?

Do: The document title, section heading and a link, so answers can cite and users can check
Do: Dates and versions, so outdated content can be filtered or flagged
Do: Permissions or owner, so retrieval can respect access control
Do: Keep tables, code blocks and list items whole where possible
Avoid: Split tables row by row without repeating the header
Avoid: Strip headings and navigation that tell the model what the chunk is about
Avoid: Reuse one chunking setup for very different content types, such as contracts and code

Frequently asked questions

What is the best chunk size for RAG?

It depends on your documents and questions. A few hundred tokens with some overlap is a common starting point for prose, but measure retrieval accuracy on real questions and adjust from there.

Should chunks overlap?

Usually a little. Overlap reduces the risk of splitting an important sentence or idea between two chunks. Too much overlap wastes storage and can return near-duplicate results.

What is semantic chunking?

Semantic chunking places chunk boundaries where the topic changes, typically by comparing embeddings of consecutive sentences, rather than at fixed sizes. It produces more coherent chunks but costs more to compute.

How do I chunk PDFs and tables?

Parse the document structure first, including headings, paragraphs and tables, then chunk by that structure. Keep each table whole or repeat its header with every part, since a table row without its header loses its meaning.

Last checked for accuracy on September 25, 2026. Written by the solidcoder team.