LLM context windows: how to budget, retrieve and evaluate

An LLM feature may work with a short question and fail when a user adds a long document or a tool returns a large result. Designing for that change requires more than choosing a model with a larger advertised context window.

The application needs a policy for what enters the request, what remains available outside it and how to respond when the necessary evidence will not fit. Context management is part of the product’s reliability and information architecture.

Understand the limits you are budgeting

A context window describes the information available to a model during a request. Token accounting can include instructions, conversation history, tool definitions, retrieved material and generated output. Exact input, output, reasoning and multimodal limits depend on the selected model and API.

Check those rules directly before implementation. Anthropic’s context-window documentation (opens in a new tab), for example, explains how request components and generated output consume capacity, including model-specific behavior. An advertised context size is not an assurance that the same number of tokens is available for documents alone.

Count representative payloads with the provider’s supported tooling. Character counts, page counts and rules of thumb about words are useful estimates, but they are not substitutes for the tokenizer and accounting used by the deployment.

Work through a concrete budget

Consider a hypothetical application budget of 32,000 tokens, chosen for illustration rather than as a claim about a particular model. Assume the deployment’s accounting includes both input and output in that total. One possible allocation is:

• 3,000 tokens for instructions and tool definitions.
• 4,000 for the current question and selected conversation history.
• 16,000 for retrieved evidence.
• 3,000 for expected tool results.
• 4,000 reserved for generated output, including reasoning where applicable.
• 2,000 of headroom for measured variation.

The allocation totals 32,000. If a tool unexpectedly returns 9,000 tokens, the application needs to select, paginate or summarize the result before the next model request. Consuming the output reserve can produce an incomplete answer even when the input was accepted.

Validate separate output and request-size limits as well. Headroom is an application design choice, and its size should follow observed payload variation rather than a fixed percentage applied to every workflow.

Choose evidence by the task

For a question about one policy clause, retrieve relevant passages with document identifiers and effective dates. For a comparison across an entire short contract, supplying the complete permitted document may be easier to evaluate than retrieving isolated fragments.

Retrieval is useful when the source collection exceeds the budget or when only a small part is relevant. It also creates its own failure modes: the needed passage may not be returned, or surrounding qualifications may be omitted. Measure retrieval coverage separately from answer quality.

Apply access controls before material enters the model request. A correct answer drawn from a document the user cannot access is still an unacceptable result. Retrieved text should remain evidence, not authority to change the application’s instructions or permissions.

Keep history useful and inspectable

Preserve current decisions, unresolved questions and references to important source material. Older conversational wording can often be summarized, but a summary should not silently replace a precise requirement or a quotation needed for the task.

Store durable records outside the conversation with appropriate access controls. A model’s context is temporary working material, not a reliable system of record. Make it possible to retrieve original evidence when a compressed account becomes insufficient.

When required information cannot fit, use an explicit fallback: process bounded sections, request a narrower question or route the task for review. Avoid truncating critical evidence while presenting the result as complete.

Distinguish prompt caching from answer reuse

Prompt caching reuses computation for an eligible repeated prompt prefix. The model still generates an output; cached context still occupies capacity. Anthropic’s prompt-caching documentation (opens in a new tab) describes prefix matching and provider-specific cache behavior. Evaluate actual hit rates and latency instead of assuming every request will benefit.

An application response cache is different: it returns a previously generated answer and may avoid a model call. Its key and invalidation policy must account for permissions, source versions and relevant request settings. Similar-looking questions are not necessarily interchangeable, particularly when the user’s access or the underlying records differ.

Evaluate how the budget affects results

The Lost in the Middle research paper (opens in a new tab) found that the location of relevant information affected performance in the models and tasks studied. Treat that finding as a reason to test the chosen deployment, not a universal performance claim about every current model.

Build an evaluation set containing long inputs, conflicting passages, evidence near the middle and questions that cannot be answered from the supplied material. Compare answer correctness, source support, retrieval coverage, latency and cost. Include the longest realistic requests and concurrent workload.

A useful design deliverable is a documented budget, evidence-selection policy, overflow behavior and repeatable evaluation. Revisit them when models, tools or source collections change.

Explore AI & Machine Learning services, or Discuss a Project with the documents, interactions and response requirements your LLM feature needs to handle.