LLM Context Windows: What They Are and Why Bigger Is Not Always Better

A context window is the total amount of text a model can consider at once — your prompt, any documents you supply, the conversation so far, and the response it is generating. Everything competes for the same budget.

Tokens, not words#

Models process tokens, which are sub-word chunks. As rough guidance for English:

• 1 token ≈ 4 characters ≈ 0.75 words
• 1,000 tokens ≈ 750 words ≈ 1.5 pages

Code tokenises less efficiently than prose — punctuation, indentation and identifiers like getUserByEmailAddress each cost more tokens than their character count suggests. A 500-line TypeScript file is often 5,000–8,000 tokens rather than the ~3,000 a word-count estimate implies.

Non-English text costs more still, sometimes two to three times as many tokens for the same meaning.

Bigger windows do not mean uniform attention#

This is the practical finding that matters most: models attend more reliably to the beginning and end of a long context than to the middle. The effect is well documented and it has a direct consequence for how you structure a prompt.

Put the instruction last. If you supply 50,000 tokens of code and then ask your question, the question is in the strong recency position. Asking first and then pasting buries the instruction.

Put the most important reference material first or last. Not in the middle of ten files.

Do not treat a large window as a substitute for relevance. Filling 200,000 tokens with everything you have is usually worse than supplying the 10,000 tokens that matter. More context is not more signal; it is also more distraction.

What long context is genuinely good for#

• Whole-file and multi-file reasoning where the relationship between files is the point — a refactor that touches five modules.
• Long documents where you cannot know in advance which part is relevant.
• Extended conversations that need to remember earlier decisions.

Managing a budget#

Send diffs, not files. For a change request, the diff plus the immediately relevant functions is usually enough and is a fraction of the tokens.

Summarise history. In a long session, a compact summary of decisions so far costs far less than the full transcript and often works better, because it removes noise.

Strip what does not help. Lockfiles, build output, minified bundles, node_modules — none of it helps and all of it competes for attention.

Use prompt caching if your provider supports it. A large, stable prefix — your project instructions, a schema, a set of reference files — can be cached so repeated requests do not re-pay for it. This changes the economics of a long system prompt substantially, and it rewards putting the stable content first and the varying content last.

RAG versus long context#

Both put relevant information in front of the model. They are not competitors so much as tools for different scales:

Retrieval searches a corpus and injects the top matches. Right when the corpus is far larger than any window — a whole documentation site, years of tickets, a large codebase. Cost and latency stay flat as the corpus grows.

Long context puts the material in directly. Right when the relevant set is small enough to fit and you cannot reliably predict which parts matter. Simpler, with no retrieval quality to tune.

A common effective pattern is both: retrieve broadly to select candidate documents, then supply the full text of the selected ones rather than fragments — retrieval for selection, long context for comprehension.

Frequently asked questions#

Does a longer prompt cost more?#

Yes, roughly linearly in input tokens, and latency grows too. Prompt caching reduces the cost of a repeated prefix substantially where it is available.

What happens if I exceed the window?#

Either an error, or silent truncation depending on the API and client. Silent truncation is the dangerous one — the model answers from partial information with no indication that anything was dropped.

How do I count tokens accurately?#

Use the provider's tokeniser or token-counting endpoint. Character-based estimates are fine for planning and unreliable near the limit.

Is a bigger context window always better value?#

No. Beyond the point where your relevant material fits, extra capacity mostly buys the opportunity to add noise. Relevance beats volume.

Related reading#

• Prompt Engineering for Developers
• AI Coding Assistants
• Counting characters for a prompt budget? The Character Counter gives an exact count including UTF-8 bytes.

References#

• Liu et al., Lost in the Middle: How Language Models Use Long Contexts
• Anthropic: Prompt caching