Building an AI agent today usually takes only a few lines of code before it can reach GitHub repositories, Slack channels, Google Drive, databases, and browser tools. But once those connections are in place, the problem changes. You may only want the agent to answer a simple question like “what went wrong with yesterday’s deployment?” Before that short user request ever reaches the model, the system may have already packed the context window with JSON descriptions for twenty tools, current database schemas, user permission parameters, the last fifty commit messages, and HTML fetched by the browser tool in the previous step.
The user’s short question gets buried under thousands of tokens of machine-generated payload. Connecting tools has become easy. Keeping the model focused on the actual problem inside all that noise has become the hard engineering work. The problem has shifted: older systems often lacked context; modern agents more often ingest too much irrelevant information.
This sounds a lot like retrieval-augmented generation, or RAG: find the relevant materials, discard the irrelevant ones, and the problem is solved. Right?
RAG only solves the first half of context governance: where to find material. It retrieves text from an external database and hands the selected chunks to the model. But once the material has been retrieved, or once the agent is actually running, the hard part has only started. Which retrieved items should enter the model window immediately? Which ones should remain in the local file system without consuming tokens? What should happen to massive debug logs produced halfway through tool execution? Which data should be compressed? Which intermediate state has expired and should be discarded? These later questions about information routing, memory management, and runtime state are outside the scope of plain RAG.
This does not mean RAG is useless. It also does not mean long-context models have lost value. RAG remains one of the most stable and basic ways to connect a model to external data. But the engineering premium of a static “chunk, embed, retrieve, concatenate” pipeline is declining. The harder problem is what happens when large context windows and many external channels are both available: how do you use a runtime management layer to protect the model’s attention, control cost, and avoid drowning the system in noise? Long context solves the physical problem of fitting more tokens. It does not automatically produce low latency, high cache hit rates, or a clear decision path. The real shift is simple: the system can know a lot, but the model should not see all of it on every turn.
Connection protocols have made tool integration unusually easy. Protocols such as Model Context Protocol, or MCP, can turn external resource access into a few lines of code. But the easier tool connection becomes, the dirtier the model’s context window becomes.
In the traditional approach, the system has to put tool definitions, parameter types, and descriptions into the prompt so that the model knows what it can call. A complex code execution toolkit may contain more than a dozen interfaces, and the tool definitions alone can consume tens of thousands of tokens. The bigger problem is what happens during execution. Debug logs, stack traces, and giant JSON arrays may keep flowing through the model’s attention mechanism across turns. This makes each call expensive and can also cause the model to lose track of the core instruction.
Anthropic explored one response to this problem in its MCP code execution practice. Instead of putting all tool descriptions into the global prompt, the agent runs inside a sandboxed execution environment where it can discover tools dynamically, run code, and filter results locally. If the full tool surface were placed into the window directly, it would consume more than 150,000 tokens. By letting the sandbox run code and handle intermediate data, the information that finally reaches the model for a given step can be reduced to about 2,000 high-signal tokens. This separates the control plane from the data plane: the connection layer opens the tool channel, while the governance layer decides which data enters the model.
In production-grade agent applications, context can no longer be treated as a string assembled by appending more text. It is a temporary compiled projection of the system’s runtime state for the current step.
To manage this layered state, Google proposes a similar structure in Google ADK. It splits agent runtime state into working context, session state, long-term memory, and artifacts. The model’s current window is only a snapshot of this deeper state for one inference step.
Under this architecture, information flow changes in two ways.
First, many intermediate artifacts and state objects can safely remain outside the window. Files, generated code, and data tables produced during a task are off-prompt by default. They live in the external execution environment as paths or references. Only when the model later decides to inspect a file does the system load the relevant part into the window.
Second, the system does not blindly append all long-term memory and session history. A governance layer filters and compresses those materials before they re-enter attention.
This runtime architecture changes when external information enters the model. It is no longer only about retrieval accuracy. It is about information routing and lifecycle management inside the whole system.
Once agents move into production, the pressure to manage context often comes from bills and latency. In complex long-horizon tasks, the ratio between input and output tokens can become extremely lopsided.
According to Manus’s engineering writeup, its production agents reached an average input-to-output token ratio of about 100:1. At that ratio, KV cache hit rates and stable prompt prefixes directly affect whether the system is economically viable and responsive enough to use. If every turn puts volatile intermediate state, timestamps, or irregular tool output near the beginning of the context, prefix caching keeps breaking. Each call has to recompute tens of thousands of tokens, driving up cost and latency.
Manus uses a strategy that combines the file system with state management.
First, the file system becomes the final context store. Original materials stay outside the window in an isolated sandbox. The model usually works with paths and recoverable references. As long as it does not explicitly read a large file, that file does not pollute the prefix cache.
Second, stable and standardized tool call formats keep the beginning of each request consistent, which helps maximize provider-side caching.
Third, long-context models often suffer from “lost in the middle,”
where information placed in the middle of a large prompt gets ignored.
Manus counters this by regularly updating and rewriting a
todo.md task board near the end of the current window,
forcing goals and key variables back into recent attention. This shows
that state governance is not an abstract architecture concern. It
directly controls system performance.
As context overload becomes a common pain point, dedicated tools for context reduction are starting to appear. Headroom is one narrow example of this direction becoming productized.
Headroom does not replace RAG or manage front-end tools. It sits between the agent application and the model API as a transparent proxy, targeting redundant payloads right before they enter the model. Long JSON responses, thousands of lines of build output, and extracted HTML from web pages can be compressed by a micro-compression engine that recognizes content types and preserves errors, important fields, and boundary data. In its public benchmarks, Headroom reports token reductions of 60% to 95% for JSON, logs, and build output in specific scenarios.
To preserve recoverability, it uses a mechanism called
compress-cache-retrieve. When Headroom compresses a segment, it leaves
behind a small retrieval marker and stores the full original text in a
local database. If the model later needs the exact error field that was
compressed away, it can call a built-in ccr_retrieve tool
to recover the original. Engineering-wise, this separates token savings
from the ability to inspect details later.
This kind of middleware is still a narrow attempt. Tools like Headroom do not yet have large-scale, systematic proof that they improve success rates for difficult programming agents. If the workload already consists of compact source code, well-filtered search results, or short everyday conversations, a compression layer may do little or even hurt end-to-end performance by disrupting provider caching or adding local database round trips. Its main significance is that it demonstrates a viable path for middleware filtering when agents handle large machine-generated payloads. It is not a universal cure.
The design center has moved. Early prompt engineering was often about writing a better instruction. Modern agent engineering is increasingly about choosing among middleware, state engines, file systems, memory layers, and execution environments. For future agent systems, success may depend less on whose model has the largest physical window and more on who can build a better information runtime.
From this view, several old instincts need adjustment.
First, RAG remains foundational. It is like SQL for external knowledge: a way to fetch data. But writing a neat RAG pipeline that concatenates retrieved chunks into a prompt is no longer enough. The higher-value work is how that retrieval layer interacts with workflows, multi-agent coordination, state, permissions, and runtime routing.
Second, the file system is not merely a place to store final outputs. Original materials can stay outside the model, while the agent works with paths and pointers inside a sandbox. The model can decide when to read, how much to read, and what to leave untouched. That is a practical way to control the timing of information entering the window.
Third, raw chat logs, intermediate tool outputs, and execution traces are not agent memory. Replaying them verbatim into future context is expensive and noisy. Useful memory has to be deduplicated, compressed, and turned into reusable operating procedures, while useless residue is discarded.
Large windows and RAG open the data channels to the outside world. That is only the first step. Without careful management at the entrance, even a strong model will become slower, more expensive, harder to audit, and less trustworthy as the surrounding information grows.