What to Do When You Hit an AI Model's Context Limit

Hitting a context limit feels like a capacity problem, so the obvious fix is a model with a bigger window. That's usually the weakest move available. In one study across 18 models, the same questions answered from focused prompts of around 300 tokens beat the same questions answered from full prompts averaging about 113,000 tokens — on every model tested. Cutting what you send isn't just cheaper than upgrading; it often produces better answers. Here's how to find what's filling your window, what to cut and in what order, how to reshape work that genuinely doesn't fit, and when a larger window really is the right call.

First, work out what's actually filling the window

The context window is one shared budget for the whole request, and most of what's in it usually isn't the user's input. Before changing anything, find out where the tokens are going — the answer is frequently surprising.

Everything in the request counts

The window holds the system prompt, every message in the conversation history including tool results, any images or documents you attach, and your tool definitions — whether or not a tool gets called. It also holds the model's own output for the turn, including any reasoning tokens it spends before answering. Input, output and reasoning all draw on the same number.

Two consequences catch people out. Tool definitions are charged to the window on every request, so a large tool catalogue is a fixed tax on every turn. And prompt caching doesn't help here at all: it changes what you pay for those tokens, not whether they occupy the window. A cached prefix takes up exactly as much room as an uncached one.

There are two different ways it fails

Knowing which failure you're looking at tells you what to fix:

The second row is the dangerous one. The request succeeds, you get a response object back, and only the stop reason tells you the answer was cut off mid-thought. Code that checks for an error and nothing else ships truncated answers without ever logging a failure. Read the stop reason on every response, not just the status code.

One more detail if you call models through a gateway: provider errors are typically passed through unmodified, so the exact wording of a context-length error comes from whichever model you called. Two models behind one endpoint will phrase the same failure differently. That's a good reason to branch on the structured error type and the stop reason rather than pattern-matching the message text — the AVIS error reference documents which errors are the gateway's own and which are passed through from the provider.

Measure before you guess

Guessing which part of the request is largest wastes an afternoon. Every generation records what it consumed, and reading those numbers takes a minute. The AVIS usage record reports promptTokens, completionTokens and totalTokens per generation, so you can compare a request that worked against one that failed and see exactly which component grew.

For a pre-flight estimate, count client-side with the tokenizer library for the model family you're calling, or send one short probe request and read the prompt token count back. Character count divided by four is a reasonable back-of-envelope figure for English prose and unreliable for code, JSON, or non-Latin scripts — which is exactly the content most likely to blow a budget. Whatever method you use, leave headroom: reserve enough of the window for the longest answer you expect, plus reasoning tokens if the model uses them.

Cut what's in the window, in this order

Work from the largest and least valuable content down. In most applications that order is tool output first, tool definitions second, old conversation turns third, and the user's actual input last.

Start with tool output

Tool results are usually the biggest and most disposable thing in an agent's window. A search that returns ten full pages, a database query that returns every column, a file read that returns a whole module — each lands in the history and stays there for the rest of the run, long after the model has extracted the one fact it needed.

Two fixes, and they compose. Trim at the source: return the fields the model needs rather than whatever the underlying API emits, and cap result sizes before they enter the conversation. Then clear stale results as the run proceeds — once a result has been acted on, replacing it with a one-line note about what it contained frees a lot of space for very little loss. Some APIs now offer this as a built-in option.

Tool definitions deserve the same scrutiny. If your agent carries thirty tools and uses three, the other twenty-seven are paying rent on every request. Loading tool definitions by task, or letting the model search for a tool instead of holding every schema in context, removes a cost you were paying continuously.

Trim history without losing the thread

For conversations, the crude fix is a sliding window: keep the last N turns, drop the rest. It's simple and works for genuinely stateless interactions like short support exchanges. It fails the moment the user says "like I mentioned earlier," because what they mentioned has been deleted.

Summarising works better where earlier turns still matter at a high level. Keep recent turns verbatim, compress older ones into a running summary, and refresh that summary on a schedule rather than every turn. Two rules matter in practice: keep anything with a commitment or a constraint in it — dates, names, decisions, preferences — because summaries lose specifics first, and keep the system prompt outside the compression so instructions never get summarised away.

Retrieve instead of sending everything

When the material is reference data rather than conversation — documentation, a knowledge base, a large codebase — selecting the relevant parts beats sending all of it. Embed and index the corpus once, then retrieve the handful of passages relevant to each question and send only those.

This is where the research gets encouraging: sending less isn't only a cost saving. Short focused prompts outscored full-length ones across all 18 models in that study, so retrieval done properly gives you a smaller window and better answers. For anything document-shaped, it's the highest-leverage change in this list.

When it still doesn't fit, change the shape of the job

Some inputs don't fit under any reasonable trimming, and at that point one long request is the wrong design. Split the work instead.

Chunking splits a long document into pieces small enough to process individually. A common starting point is roughly 400–512 tokens per chunk with 10–20% overlap — about 50 to 100 tokens of overlap on a 500-token chunk — with smaller chunks for narrow factual lookups and larger ones for questions that need extended reasoning. Overlap exists so a sentence that straddles a boundary still appears whole somewhere. Split on structure where you can: headings, paragraphs and function boundaries make better edges than a fixed character count.

Treat those numbers as a starting point rather than a setting. Reported recall differences between the best and worst splitting strategies on the same content run to around nine percentage points, which is large enough to be worth an afternoon of measurement on your own documents and small enough that nobody should rewrite their pipeline on someone else's benchmark.

Once the content is chunked, two patterns cover most work:

• Map-reduce processes every chunk independently and then combines the results. It's fast because the chunks are parallel, and it's the right choice for extraction, classification and summarisation, where each piece can be understood on its own.
• Refine passes an answer through the chunks in sequence, updating it as it goes. It's slower and sequential, but it holds a thread across the document, which matters for narrative or argumentative content.

Both break on the same thing: a question whose answer depends on two facts that land in different chunks. Nothing in the pipeline sees both at once, and the model will confidently answer from the half it can see. Where that risk is real, either retrieve with enough breadth that related passages arrive together, or run a second pass over the intermediate results where the facts can finally meet.

A bigger context window is a real option, and a weaker one than it looks

Moving to a model with a larger window is legitimate, and sometimes correct. It just doesn't deliver what most people expect, for three separate reasons.

Position matters more than capacity. Peer-reviewed work on long contexts found a U-shaped accuracy curve: models do best when the relevant information sits near the beginning or the end of the input, and noticeably worse when it's buried in the middle. That result held even for models explicitly built for long contexts, which is the part that should change how you plan. A window twice as large doesn't mean twice as much usable attention.

Degradation starts well before the limit. Accuracy and recall fall off gradually as input grows rather than holding steady until a cliff. The effect is uneven — it depends on how closely the question matches the wording of the relevant passage, and on whether similar-looking but wrong content sits nearby. In testing across 18 models, even a single distracting passage measurably reduced accuracy compared with a clean prompt.

Long prompts cost more and run slower on every request. A conversation that grows without bound is a bill that grows with it, because the entire history is re-sent and re-charged every turn. Model prices are per token, and through a gateway you also pay the platform's share on that larger number — AVIS passes each model's price through unchanged and adds a 10% platform fee on API usage, so a window you didn't need is a fee you didn't need either.

When is a bigger window actually right? When the task needs reasoning across a whole document at once and chunking would destroy the relationships that matter — a contract where a clause on page 40 modifies one on page 3, a change that touches a dozen files, a transcript where the answer is the arc rather than a fact. There, the window does real work instead of absorbing content you could have filtered.

If you do switch, make the switch cheap: keep the model name in configuration rather than in code, and use an endpoint that reaches several providers so testing a larger-window candidate doesn't mean a new integration. One thing to check directly with the provider — catalogue endpoints often list a model's supported parameters without listing its context window. The AVIS model list exposes capabilities.text.supportedParameters and per-parameter bounds, so confirm the window size against the provider's own documentation before you commit.

FAQ

What does "maximum context length exceeded" mean?

It means the request didn't fit in the model's working memory. That budget covers the system prompt, the whole conversation history, tool definitions and tool results, any attached images or documents, and the response the model is about to generate. The error usually means your input alone was too large, but the same limit can also stop a response partway through generation.

Does the output count toward the context window?

Yes. Input, output and any reasoning tokens the model spends share one budget. That's why leaving headroom matters: if you fill the window with input, there may be no room left for a complete answer, and the response will stop early rather than fail cleanly.

How many tokens is 1,000 words?

Roughly 1,300 to 1,500 tokens for ordinary English prose, but treat that as an order of magnitude rather than a number to budget against. Code, JSON, tables and non-Latin scripts tokenize far less efficiently, sometimes at more than double the rate. Count with the tokenizer for the model family you're calling, or read the prompt token count back from a real request.

Should I summarize or truncate conversation history?

Truncate when older turns genuinely don't matter, such as short transactional exchanges. Summarize when they do, keeping recent turns verbatim and compressing older ones into a running summary. Either way, keep the system prompt out of the compression, and preserve specifics like dates, names and decisions, since those are what summaries lose first.

Will a larger context window fix my problem?

Sometimes, but it's usually the last thing to try rather than the first. Accuracy degrades gradually as input grows, models retrieve information from the middle of a long input less reliably than from the start or end, and every extra token costs money and time on every turn. Reduce what you send first; move to a larger window when the task genuinely needs the whole document at once.

Does prompt caching free up context window space?

No. Caching changes what you pay for repeated tokens, not whether those tokens occupy the window. A cached prefix takes the same amount of room as an uncached one, so caching is a cost optimisation rather than a capacity one.

Wrapping up

When you hit a context limit, resist the upgrade instinct. Read the usage numbers to find out what's actually filling the window, then cut in order: tool results first, then unused tool definitions, then old turns via summarisation, then retrieval instead of stuffing whole documents. If it still doesn't fit, chunk it and pick map-reduce or refine based on whether the content needs a thread running through it.

Save the bigger window for work that genuinely needs the whole thing in view at once, and keep the model name in configuration so that choice stays cheap to revisit. If you'd like to compare candidates without building a separate integration for each, AVIS puts models from several providers behind one API key with OpenAI-compatible and Anthropic-compatible endpoints, and records the token count and cost of every generation so you can see the effect of each cut.

Last updated: September 2026