How to choose chunk size with evals
Chunk size decides how much text the model reads in one call, and it changes quality even when the whole document would fit in the context window. For fixed-output tasks, where the answer stays short however long the input is, use the largest chunk likely to contain the answer. For expansive-output tasks, where the output grows with the input, use smaller chunks processed one by one and combined, and in both cases, try three or four sizes on the same eval set and let the pass rate decide.
Why chunk size matters even with long context
Modern models accept very long inputs, so it's tempting to send whole documents. But the "Lost in the Middle" study (2023) found models use information at the start and end of a long input much better than information in the middle. Accuracy followed a U shape: highest when the key information sat at the start or the end, lower when it sat in the middle, even for models built for long contexts.
So chunk size isn't only a retrieval setting. It decides what the model pays attention to.
Two kinds of document tasks
Husain and Shankar split document tasks into two types, and the right chunk size is different for each.
Fixed-output tasks
The answer stays short no matter how long the input is. "What is the penalty clause in this contract?" "What was the net revenue in Q3?"
Use the largest chunk likely to contain the answer. Fewer calls, and less chance of splitting the answer across two chunks.
If that gets too slow or too expensive, filter the document first, with keyword search or a small retriever, to find the relevant section. Don't shrink the chunks.
Expansive-output tasks
The output grows with the input. "Summarise each section." "List every customer complaint in these 200 emails."
Smaller chunks work better here. Process each chunk on its own and combine the results, a pattern called map-reduce: map a prompt over every chunk, then reduce the results into one output. A single call over a huge input tends to skip items, especially in the middle.
fixed output: [ one large chunk ] ──> short answer
expansive output: [c1] [c2] [c3] [c4] [c5]
│ │ │ │ │ (in parallel)
└────┴────┼────┴────┘
▼
merged long list
Where to cut
Cut chunks at natural breaks, like paragraphs, sections or clauses, so one idea isn't sliced in half. A clause split across two chunks can be retrieved half-way and misread. See chunking strategies.
Cost of small chunks
Small chunks lose the big picture. An answer that depends on two parts of a document, like a definition in section 1 and an exception in section 9, can fail when each chunk is read alone. Add test cases where the answer depends on two chunks together, so your eval catches this.
Tune it like any setting
Treat chunk size as a setting to tune, not a constant from a tutorial.
- Pick three or four sizes, for example 500, 1000, 2000 and 4000 tokens, with your normal overlap.
- Run the same eval set on each.
- Compare pass rate, recall@k, cost and latency.
- Choose, and re-test when you change models or document types.
| Chunk size | Recall@5 | Answer pass rate | Cost per 1,000 questions | p95 latency |
|---|---|---|---|---|
| 500 | ||||
| 1000 | ||||
| 2000 | ||||
| 4000 |
From the floor
In the financial report RAG, we used chunks of 1000 characters with an overlap of 200, retrieved the top 5 chunks, and answered with Llama 3.1 running locally through Ollama.
In the Build30 RAG session, a builder asked how to choose a chunking strategy when there are so many, overlapping and others, and my first answer was that it depends on the type of queries you're building for. This play turns that answer into a test: the type of query decides whether you're in a fixed-output or an expansive-output task, and the eval set decides the size.