Context windows: what fits, and why more isn't always better

What a context window is

An LLM has no memory between calls. Every time you send a request, the model reads the whole input from scratch and generates a response, and then it forgets everything.

The context window is the size of that input-plus-output space, measured in tokens. It is like a box: everything you want the model to consider in this call has to fit inside it, and the answer has to fit in the space that is left.

Context windows have grown fast. Early GPT-3 models had about 2000 tokens, and many current models accept hundreds of thousands, with some offering a million or more. Most models also have a separate, smaller limit on how many tokens they can generate in one response, so check both numbers for the model you use.

What fills the context window

When you build an application, much more than the user's message goes into the window.

Part	Example	How big it can get
System prompt	Instructions, rules, persona, examples	Hundreds to thousands of tokens
Tool definitions	Names, descriptions and schemas of every tool an agent can call	Grows with every tool you add
Conversation history	Every earlier user message and model reply	Grows every turn
Retrieved context	Chunks from your documents in a RAG system	Depends on how many chunks you add
Tool results	Search results, database rows, file contents	Can be huge if not trimmed
The answer	What the model writes	Up to the output limit

In agents especially, the history and tool results grow with every step, and that is where most context problems in production come from.

Why a chatbot "forgets"

When a long conversation with a chatbot starts losing track of something you said at the beginning, two things can be happening.

The first is that the earlier messages are no longer in the window. Because the model has no memory, the application sends the whole conversation again on every turn. When the conversation gets too long to fit, the application has to drop or summarise older messages, and whatever was dropped is gone for the model.

The second is less obvious: the information is still in the window, but the model doesn't use it well. This is covered in the next section, and it matters more than most people expect.

Advertised context and effective context

A model accepting 200000 tokens means you can send 200000 tokens without an error. It doesn't mean the model pays equal attention to all of them.

Research has shown this several times. In 2023, researchers found that models answer questions much better when the relevant information is at the beginning or end of a long input than when it is in the middle, a pattern they called "lost in the middle". In 2025, Chroma tested 18 models and found that performance became less reliable as input length grew, even on simple tasks, and that adding similar but irrelevant text, distractors, made it worse, which they called context rot.

You may also see "needle in a haystack" results, where a model finds one planted sentence in a long document almost perfectly. Those tests are useful but easy, because the needle is usually very different from the haystack. Real tasks, like finding the one clause that applies among fifty similar clauses, or combining facts spread across a long document, are much harder.

So treat the advertised context window as a ceiling, not a target, and test the model at the input lengths and with the kind of documents your application will actually send.

Long context costs money and time

Every token in the window is processed on every call.

• Cost: you pay for input tokens each time. If every turn of a chat resends 30000 tokens of history, every turn costs 30000 tokens of input, even if the new message is one line.
• Latency: before generating the first token, the model has to process the whole input, a step called prefill, so long inputs mean a longer wait before anything appears.
• Memory: the model keeps intermediate results for every token in a structure called the KV cache, so long contexts need much more GPU memory, which is part of why long-context requests are priced higher or limited.

The pricing page shows how quickly resending history adds up.

How to manage context in your application

Good applications decide what goes into the window instead of filling it. The broader practice is called context engineering, and these are the basic techniques.

Keep the fixed parts small

Review your system prompt and tool definitions regularly. They are sent on every call, so an extra 2000 tokens of instructions that the model doesn't need costs you on every request and adds more text for the important parts to compete with.

Trim and summarise history

For conversations, keep the recent turns in full, and replace older turns with a short summary of what matters: the user's goal, decisions made, facts established. For agents, store long tool results outside the context and keep only what the next step needs, like the 5 relevant rows instead of the whole table.

Retrieve instead of stuffing

If you have 500 pages of documentation, don't put all of it in the window because it fits. Use retrieval to find the few chunks that answer the question. It is usually cheaper, faster, and often more accurate, because there is less irrelevant text to distract the model. Long context vs retrieval covers when each one wins.

Put important information where the model uses it best

Put the instructions and the most important information near the start or the end of the input, not buried in the middle of a long document. Many prompts work better when the documents come first and the question or task comes last.

Structure the context

Separate parts clearly, with headings or tags like <documents> and <question>, label each retrieved chunk with its source, and remove duplicates. Clear structure helps the model find what it needs, and helps you debug what it saw.

Reuse stable prefixes with caching

If many requests start with the same long text, like a big system prompt or a document many users ask about, prompt caching lets the provider reuse the processing of that prefix, which cuts cost and latency. It works only when the repeated part is exactly the same and comes first. See prompt caching.

Break big tasks into smaller ones

For coding agents and long agent runs, one huge conversation slowly fills with old instructions, outputs and errors, and the agent starts missing earlier instructions. Give the agent smaller tasks, one component at a time, start fresh contexts for new tasks, and keep important decisions in a file the agent reads, instead of hoping it remembers them from 50 turns ago.

Long context is still useful

None of this means long context is bad. It is very useful when the task needs the whole document at once, like summarising a long report, comparing two contracts, or reviewing a whole codebase module, and when retrieval would risk missing something.

The point is to choose it deliberately, test that the model uses what you send, and know what it costs.

How to test effective context for your use case

1. Take real documents from your application, not generic text.
2. Write questions whose answers are at different positions: the start, the middle and the end.
3. Run them at different total input lengths, for example 5000, 30000 and 100000 tokens.
4. Add realistic distractors, like similar documents that don't contain the answer.
5. Measure accuracy, latency and cost at each length.

The length where accuracy starts dropping is your effective context for that task, and it is the number you should design around.

From the floor

In one of my LLM lectures, I said ChatGPT will always remember what you discussed, however long the conversation gets. That explanation was wrong. A model has a limited context window, and even when information fits inside it, that does not mean it will use every detail correctly.

I see the same thing when I work with Claude Code on a project. As we keep working and the context fills up, it can miss some of the earlier instructions, so I give it smaller tasks, one component at a time, and check what it changed before moving forward.

Related

• Context engineering
• Long context vs retrieval
• LLM pricing math
• KV cache

Sources

• Lost in the Middle: How Language Models Use Long Contexts, Liu and others (2023)
• Context Rot: How Increasing Input Tokens Impacts LLM Performance, Chroma (2025)