The Context Window and Long Documents: Token Limits, Lost in the Middle, Long Context vs RAG

There's a limit to how much text you can feed a language model — and even when you stay under it, the model tends to overlook information stranded in the middle of the context far more easily than information at the edges. In this article we explore the context window with everyday analogies, examining the token limit, the "lost in the middle" problem, the choice between long context and RAG, and summarization chains.

What is a context window?

When you talk to a language model, the model only "sees" the text handed to it at that moment: your system instructions, the prior messages, and the question you asked. The entire set of text the model can read and reason over in a single pass is called the context window. Its size is fixed and is measured in tokens.

Think of it like a person's desk. Only so many sheets of paper fit on it at once. To add a new document when the desk is full, you have to remove an old one. The context window works exactly this way: the model can hold as much text as its "short-term memory" allows at once; anything outside that limit becomes invisible.

The model "knows" nothing outside its context window; it reasons only with whatever is inside that window.

An important distinction: the context window is not the general knowledge the model learned during training. Training knowledge is baked into the model's weights and is permanent. The context window, by contrast, is specific to the current conversation, temporary, and refilled from scratch on every request.

The token limit and why it exists

A token is the smallest unit the model uses when processing text; it's roughly a fragment of a word. In English, an average word takes about 1.3 tokens; in languages with rich suffixes (like Turkish) that ratio is usually higher. The context window is measured in this token count, and both the input (prompt) and the output the model generates share this budget.

So why is there a limit? Because the attention mechanism in the Transformer architecture compares every token with every other token. If the number of tokens is n, the number of these comparisons grows roughly with n². In other words, doubling the context makes the compute and memory cost roughly quadruple.

Context = 1,000 tokens   -> ~1,000,000 pairwise relations
Context = 2,000 tokens   -> ~4,000,000 pairwise relations  (4x)
Context = 10,000 tokens  -> ~100,000,000 pairwise relations (100x)

# cost ~ n^2  =>  the bigger the context, the faster cost climbs

The "lost in the middle" problem

Even when the window is large, the model is not guaranteed to pay equal attention to every token. Research has shown that models use information at the beginning and the end of a long context far more reliably than information in the middle. This tendency is known in the literature as the "lost in the middle" problem.

A familiar analogy: picture a long meeting. People usually remember what was said first and what was said last; the discussion in the middle slips onto the dusty shelves of memory. Watching a series, the first and last episodes stick more than the ones in between. Language models experience a similar "edge effect."

The practical consequence is clear: a critical sentence buried in the exact middle of a document can slip past the model even if it fits in the window. That's why placement matters when working with long contexts.

• Put the most important instructions and information at the beginning or the end of the context.
• Repeat the question right after the relevant documents — that is, near the end.
• Avoid padding the window with irrelevant text; noise drowns the real signal.

A large window does not mean the model will read everything you put in it with equal seriousness.

Long context or RAG?

Say you're working with a collection of documents running into hundreds of pages. There are two basic paths. The first is to fit as much text as possible directly into a long context window. The second is to find only the pieces most relevant to the question and hand those to the model; this is called RAG (Retrieval-Augmented Generation).

The analogy: to answer a question, you either read the entire book cover to cover (long context), or you go to the index, jump to the relevant pages, and read only those (RAG). The second is almost always faster and cheaper; it also reduces the "lost in the middle" risk, because you give the model only focused, relevant pieces.

A rough comparison:

• Long context — Pros: Simple to set up; can catch subtle links across documents; needs no extra infrastructure.
• Long context — Cons: High cost and latency; lost-in-the-middle risk; the window limit can still be exceeded.
• RAG — Pros: Scalable (millions of documents); cheap and fast; grounds the answer in sources; easy to work with fresh data.
• RAG — Cons: Depends on retrieval quality; requires chunking and indexing infrastructure; if the wrong piece is fetched, the answer degrades too.

Summarization chains

What if the document doesn't fit in the window at all? When you need to summarize a text too long to process in a single request, summarization chains come into play. The core idea is to split the text into pieces and merge the summaries step by step.

There are two classic patterns. The first is map-reduce: you summarize each piece separately (map), then combine those intermediate summaries into a single whole to produce the final summary (reduce). The second is refine (iterative refinement): you start with the summary of the first piece and, as you read each subsequent piece, update the summary as you go.

# Map-Reduce summarization (pseudocode)
pieces = split_text(document, size=3000)

partial_summaries = []
for p in pieces:                       # MAP
    partial_summaries.append(summarize(p))

final_summary = summarize("\n".join(partial_summaries))   # REDUCE
return final_summary

Think of it like splitting a thick report among a team: each person summarizes their own section (map), then an editor gathers those summaries into a single executive summary (reduce). The refine pattern, instead, is like a single reader who revises and updates their notes with each new section.

• Map-reduce: Fast because pieces can be processed in parallel; but links across pieces may weaken.
• Refine: Preserves sequential flow and context better; but it's slow because it is sequential.

In both methods some information loss is inevitable; a summary is, by nature, a lossy compression. So when critical details must be preserved, combining summarization with RAG (pulling a detail absent from the summary straight from the source when needed) is a more robust approach.

A practical decision guide

Which approach to choose depends on data size and the task. A simple intuition:

1. A few pages, single document: Put it directly in the context window. The simplest and usually most accurate path.
2. Large but static collection, targeted questions: Use RAG. It scales, it's cheap, and it grounds the answer in sources.
3. Grasping the whole of one long document (e.g., "summarize this report"): Use a summarization chain.
4. Tasks that must be both broad and accurate: Use RAG + long context together.

In short, the context window is the most underestimated constraint when working with language models. The right strategy isn't to buy a bigger window; it's to concentrate the signal, place information wisely, and apply RAG and summarization where they fit. If you're curious how we put these approaches into practice in production systems, take a look at the AI architecture behind EcoFluxion.