The Attention Memory Divide and What Your LLM Context Window Actually Does

Every major AI vendor advertises context window size as a capability metric. 128K tokens. 200K tokens. 1 million tokens. The implicit promise is that the model reads and uses everything you put in that window equally. It does not. A 2023 Stanford and UC Berkeley study measured exactly how unevenly attention is distributed, and the numbers matter if you are building anything in production.

In my opinion, the research is clear on this. Attention in decoder-only transformer architectures is not uniform. Information at the beginning and end of the context receives disproportionate attention. Information buried in the middle is systematically underweighted, and in some cases produces outputs worse than if you had provided no context at all.

That is not a bug that future model versions will fix. It is a structural property of the architecture.

What the research actually shows

Liu et al. (2023) ran a straightforward test. They gave models multi-document question-answering tasks and varied the position of the document containing the answer. When the answer document was placed at the beginning or end of the context, models performed well. When it was placed in the middle, GPT-3.5-Turbo's performance dropped by more than 20 percentage points, falling below its closed-book baseline of 56.1%.

To be specific: the model performed worse with the relevant document present than it would have without any context at all. The additional context was not neutral. It was actively harmful.

The performance curve is U-shaped. High at the start of the context, high at the end, low in the middle. Researchers call this the primacy-recency bias, and it appears across model families, model scales, and task types. It is not a GPT artifact. It is an architectural pattern in decoder-only transformers broadly.

What the context window actually is and why tokenization matters

Before you can reason about what fits in the window, you need to understand how text enters it. Models do not process words. They process tokens, which are subword units produced by a byte pair encoding (BPE) algorithm first described by Sennrich, Haddow, and Birch in their 2015 ACL paper on neural machine translation of rare words with subword units (arXiv:1508.07909).

BPE starts from individual characters and iteratively merges the most frequent adjacent pairs until it reaches a fixed vocabulary size. The result is a vocabulary of subword units that balances coverage of common words as single tokens with the ability to compose rare words from smaller pieces.

The practical implication is that token count is not the same as word count, and small surface differences in text produce different token boundaries. The phrase AI infrastructure tokenizes as three tokens in GPT-4o's cl100k_base vocabulary. The same phrase lowercased to ai infrastructure tokenizes as three tokens but different ones, because the capitalized AI is a single high-frequency token while lowercase ai resolves differently. Add a hyphen and write AI-infrastructure and the hyphen itself becomes a token boundary, expanding the count. Code blocks, URLs, and structured data like JSON expand token counts faster than prose, often at two to three times the ratio.

This matters because your system prompt, your retrieved documents, and your user message all compete for the same finite token budget. A 128K token window does not mean 128K words of usable context. It means roughly 90K to 100K words of prose, less if your retrieved chunks contain code or structured data, and less again because the attention problem means only a fraction of that is reliably used by the model.

The attention problem in numbers

Lost in the Middle (Liu et al., arXiv:2307.03172) tested models on multi-document question answering where the answer was placed at different positions in a context of 10 to 30 documents.

At position 0, the first document slot, retrieval accuracy ran at roughly 80%. At the end of the context it dropped slightly but held in the mid-70s. At middle positions in a 20-document context, accuracy fell to around 50%. That is a 30 percentage point gap between best and worst positions in the same context window with the same model receiving the same relevant document.

If your RAG pipeline injects 10 retrieved chunks and places the most relevant one at position 5, you have already accepted a 25 to 30 percent accuracy penalty before the generation step even starts. The reranker, the embedding model, the vector similarity search, none of that matters if you then place the winning document in the wrong position in the final context.

Liu et al. also showed that increasing from 20 to 50 retrieved documents improved GPT-3.5-Turbo performance by only 1.5%, while latency and cost scaled proportionally. The 30 additional documents contributed almost nothing. The effective fix is not more context. It is better positioning within the context you already have.

The gap between marketed and effective context

In my experience, mPT-7B-storywriter was marketed with an 84K token context window. In LongEval benchmark testing, it achieved 50% retrieval accuracy at 16K tokens. One fifth of its advertised capacity, and at accuracy levels that would make a production system unreliable.

This is not an outlier. ChatGLM2-6B claimed 8K token support and achieved 46% retrieval accuracy at 6K tokens. The pattern across open-source models is consistent: marketed context windows are measured by what the model can process without breaking, not by what it can reliably use to produce correct outputs.

The 2024 InfBench study (Zhang et al., arXiv:2402.13718) extended this analysis with the first benchmark averaging above 100K tokens per test case and found that state-of-the-art long-context models still required significant improvement to process these lengths effectively. Extending the context window does not automatically extend reliable recall across that window.

Extended-context versions of the same model, GPT-3.5-Turbo 16K versus GPT-3.5-Turbo 4K for instance, show nearly identical performance on tasks that fit within both windows. You pay for the larger context but do not get meaningfully better outputs when your actual task fits in the smaller one.

Architecture determines outcomes, not token count

Not all architectures have this problem to the same degree. Encoder-decoder models, the T5 family and similar bidirectional architectures, show a much flatter performance curve across context positions. Flan-UL2 and Flan-T5-XXL showed only a 1.9 percentage point gap between best and worst performance positions when evaluated within their training-time sequence length.

Decoder-only models, the architecture behind GPT and most frontier models, show 10 percentage point gaps or higher. The same research showed that Claude 1.3 performed nearly perfectly on synthetic key-value retrieval at the same task where GPT-3.5-Turbo achieved only 45.6% accuracy in the worst-case middle position.

Instruction fine-tuning reduces variance slightly, from roughly a 10-point gap to a 4-point gap on MPT models, but it does not eliminate the U-shaped curve. The architectural bias persists through fine-tuning.

What each major model does differently

The three frontier models you are most likely building on handle long context differently at the architecture level, though the U-shaped attention problem does not disappear in any of them.

Claude was trained with specific attention to long-context retrieval. Anthropic has not published the full methodology but released evaluations showing that Claude 1.3 dramatically outperformed GPT-3.5-Turbo on middle-position retrieval tasks in the Liu et al. study. Newer Claude models carry that architectural work forward. The positional sensitivity curve is shallower but still measurable.

Gemini 1.5 and 2.0 use a Mixture-of-Experts architecture that routes different inputs to specialized sub-networks. Google's published evaluation showed Gemini 1.5 Pro finding embedded text 99% of the time on the Needle in a Haystack benchmark at contexts up to 1 million tokens. That result comes from a structural redesign using sparse expert routing, not from a larger standard attention window.

GPT-4o operates at 128K tokens with standard dense attention. Community benchmarks and the Liu et al. data both document positional sensitivity in the GPT family. The practical range where you get reliable middle-context retrieval is measurably smaller than 128K, depending on task complexity and document density.

The lesson across all three is that you cannot assume uniform attention because a model supports a large context. You need to test position-specific retrieval accuracy in your actual pipeline with your actual data.

Prompt caching changes cost but not where attention lands

Both Anthropic and OpenAI now offer prompt caching, and it is worth understanding exactly what it does and does not fix.

Anthropic's implementation uses a cache_control parameter with a default 5-minute TTL. Cache reads cost 0.1x the base input token price, making repeated long-context calls roughly 90% cheaper on the tokens that hit the cache. OpenAI's caching activates automatically on prompts longer than 1024 tokens and caches the key-value tensors from the model's attention layers generated during the prefill phase.

I have seen this firsthand: what caching stores is the intermediate attention computation, the KV tensor state at each layer. On a cache hit, the model skips recomputing those attention scores. This is why cached calls are faster, not just cheaper.

What caching does not fix is where attention lands within that computation. The first call, the cache write, processes the full context through the attention mechanism exactly as it would without caching. The U-shaped bias applies in full. On subsequent cache hits, the model uses the same KV tensors, which means the same attention distribution. Caching locks in whatever positional bias was present on the first call. It does not correct it.

If you have a 50,000-token system prompt that you cache across thousands of requests, you are efficiently reusing an attention computation that may systematically underweight information in the middle of that prompt. The caching gives you a cost reduction. It does not give you a retrieval improvement.

What to do about it with measurable outcomes

Retrieval-augmented generation systems are the most directly affected by positional bias. These four strategies have measurable impact and can be validated in your pipeline before you assume they are working.

Front-load critical context. Put the most important retrieved chunk at position 0 to 2 in your context, not position 5 to 8. Based on the Lost in the Middle data, this alone produces 20 to 30 percentage point retrieval improvement for the critical document. If you have one document that must be used correctly, it goes first.

Chunk to 512 to 768 tokens per document. Longer chunks span more of the context window. A 2,000-token chunk occupies space that crosses the low-attention middle zone regardless of where the chunk starts. Smaller chunks give you more precise control over what occupies which positions. They also produce better embedding quality during indexing, which improves retrieval precision before context injection even happens.

Rerank before injection and put the winner first. A cross-encoder reranker or BM25 hybrid reranking pass runs after your initial vector retrieval. The output is a sorted list by relevance. Put the highest-scoring document at position 0. Put the second-highest at the end. Fill the middle with everything else. Use RAGAS faithfulness as your measurement: it scores on a 0 to 1 scale how many of the model's generated claims can be directly traced back to the retrieved context. A typical RAG pipeline without position optimization scores 0.60 to 0.70 on faithfulness. After reranking plus front-loading, 0.80 to 0.85 is achievable without changing the model or the retrieval index.

Always test with your production system prompt included. The system prompt consumes token budget before any retrieved context appears. A 2,000-token system prompt shifts every injected document two thousand tokens further into the window. Test position-specific retrieval accuracy with the system prompt as prefix. The numbers you get without it are not the numbers you will see in production.

The memory taxonomy that helps

Researchers frame LLM memory across three distinct types. Working memory is the context window, what the model can actively process in a single call. Episodic memory is external retrieval, what you pull from a vector store or document database and inject. Semantic memory is the model weights, what the model learned during training and cannot update at inference time.

The attention-memory divide is a working memory problem. The context window is large but not uniformly accessible. The implication is that episodic memory systems, well-designed retrieval, not larger context, are the correct layer to invest in when you need reliable recall of specific information.

I have built systems that address this through structured retrieval with position-aware reranking, where the architecture compensates for the model's positional bias rather than trying to overcome it. The approach is not complex. It requires understanding that context position is an engineering decision with measurable consequences.

What to do with this today

If you are using a RAG system and getting inconsistent results, run a position test before anything else. Take a document you know contains the answer to a test question. Place it at the start of the context. Then place it in the middle. Then at the end. Measure accuracy at each position. If you see a 15 to 20 point accuracy swing, you have a position bias problem, and adding more context will make it worse, not better.

If you are evaluating models for a production system, test middle-context retrieval directly. Do not rely on the benchmark the vendor publishes. Their benchmarks measure perplexity or aggregate accuracy, not position-specific recall, which is the thing that will break your system in production.

Smaller open-source models, 7B parameters and below, show recency bias only, no primacy bias. They preferentially use information at the end of the context and largely ignore the beginning. If you are running local inference at that scale, front-load your most important instructions and put retrieved context after them, not before.

The context window is a tool with uneven capability across its length. Build with that fact, not against it. The measurement discipline that matters is not how many tokens your model accepts. It is how many tokens your model reliably uses, at each position, on your specific task, with your actual production data.