Chunking strategies for RAG and how to choose chunk size
Chunking splits documents into the pieces your RAG system retrieves. A chunk has to be small enough that its embedding represents one idea and retrieval is precise, and large enough to make sense on its own and contain a complete answer. Start with structure-aware splitting (by headings and paragraphs) at around 200 to 500 tokens with a small overlap, add the document title and section heading to each chunk, and then choose the final size by testing a few options on your own questions. There is no universal best chunk size, it depends on your documents and the questions people ask.
Why we chunk at all
There are two reasons.
The first is retrieval precision. An embedding is one vector for a piece of text. For a whole 30 page document, that vector is an average of everything in it, and a question about one clause on page 17 won't match it well. For a paragraph, the vector represents that paragraph's meaning, so the right paragraph can be found.
The second is the context window and cost. You want to send the model the few paragraphs that answer the question, not whole documents.
Chunk size trade-off
Small chunks are precise, but they can cut an answer in half or lose the context that makes a sentence meaningful. "It must be submitted within 30 days" is useless if the chunk doesn't say what "it" is.
Large chunks keep context, but they dilute the embedding, retrieve more irrelevant text, and cost more tokens per retrieved chunk.
Chunking is about finding the size and boundaries where each chunk is one coherent piece of information.
It depends on the use case
The right strategy depends on what users will ask and what the answer needs.
For simple factual questions over short policy text, like refund rules or office hours, small chunks of one or two sentences or a short paragraph work well, because each answer is one or two sentences. For educational or explanatory content, where the answer needs depth, chunks of a full paragraph or a section give the model enough to explain properly. For contracts, the natural unit is the clause. For code, it is the function or class.
Strategies
Fixed-size
Split every N characters or tokens, with some overlap. It is simple and predictable, and it ignores meaning, so it often cuts sentences, tables and lists in the middle. Fine for a first prototype, rarely the best.
Recursive
Split on the largest natural boundary first (sections, then paragraphs, then sentences, then words) until each piece is under the size limit.
LangChain's RecursiveCharacterTextSplitter works this way, and it is a good default.
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
separators=["\n\n", "\n", ". ", " "],
)
chunks = splitter.split_text(open("policy.txt").read())
Note that chunk_size here is in characters, not tokens, unless you configure a token-based length function, and 1000 characters is roughly 200 to 250 English tokens.
Structure-aware
Split along the document's own structure: Markdown headings, HTML sections, numbered clauses, slides, table rows. If your parser kept the structure, this is usually the best strategy, because authors already grouped related information. Keep the heading path with each chunk.
Sentence-based
Split into sentences, then group sentences up to a size limit. Good for short, factual text like FAQs and policies.
Semantic
Embed each sentence and start a new chunk where the meaning shifts, where the similarity between neighbouring sentences drops. It can produce more coherent chunks for unstructured text, at extra cost during ingestion, and the gains vary, so test it against simpler methods.
Code and tables
Split code by function and class, never in the middle of a function. Keep tables whole when possible, or chunk by row with the headers repeated, as described on the parsing page.
Overlap
Overlap repeats the end of one chunk at the start of the next, so information near a boundary appears in both. 10% to 20% of the chunk size is a common starting point. It reduces cut-off answers, and it increases the index size and can return near-duplicate chunks, so don't overdo it, and with good structure-aware splitting you need less of it.
Give every chunk its context
A chunk retrieved alone has lost its surroundings. "The limit is ₹50,000 per year" doesn't say which policy, which employees, or which year.
Simple fixes go a long way:
- Prepend the document title and section heading to the chunk text before embedding.
- Store the page number, document date and source in metadata for citations and filtering.
A further step is contextual chunking: for each chunk, use an LLM to write one or two sentences that situate it within the whole document, and prepend that to the chunk before embedding and keyword indexing. Anthropic reported that this approach, which they called contextual retrieval, reduced the rate of failed retrievals (in the top 20 chunks) by 35% with contextual embeddings alone, by 49% when combined with contextual keyword search, and by 67% when reranking was added, in their tests. It costs one LLM call per chunk at ingestion time, which prompt caching makes much cheaper.
Small to big
You can also decouple what you search from what you send. Index small chunks, like sentences or short paragraphs, for precise matching, and when one matches, send the model the larger parent section it belongs to. This is called parent-document or small-to-big retrieval, and it gives you precision in search and context in generation.
How to choose chunk size
Don't guess, test.
- Build a small evaluation set: 30 to 50 real questions, each with the passage that answers it.
- Index your documents with 3 or 4 configurations, for example 256, 512 and 1024 tokens, with your chosen strategy.
- For each configuration, measure how often the right passage is in the top 5 results (recall at 5).
- Run the full pipeline on the best two and compare answer quality.
- Pick the configuration that gives the best answers at an acceptable cost.
The RAG evaluation page shows how to calculate these metrics.
Common mistakes
- Chunking before cleaning, so headers, footers and page numbers end up in every chunk.
- Splitting tables and numbered lists in the middle.
- Chunks without document titles or headings, so retrieved text has no context.
- Using a chunk size larger than the embedding model's input limit, so the end of each chunk is silently cut off.
- Changing chunking without re-running the evaluation.
From the floor
On Build30 day 9, a builder asked how to choose a chunking strategy when there are so many. My answer was that it depends on the queries you are building for. For simple data like refund policies, where users ask short questions and expect one or two line answers, we kept chunks to one or two sentences, splitting on full stops, exclamation marks and question marks. For educational content, chunks should be paragraphs, because the answer needs depth to be useful to the learner.
For the financial report project, we used LangChain's recursive splitter with chunks of 1000 characters and an overlap of 200, which is a sensible starting point for long reports, and the starting point is what you then test against your questions.