New "SELFCOMPACT" Method Empowers LLM Agents to Intelligently Manage Context Windows and Reduce Costs

A team of researchers has introduced a new inference-time framework called SELFCOMPACT, designed to help large language model (LLM) agents manage their growing context windows by deciding when and how to compact accumulated information. Authored by Tianjian Li, Jingyu Zhang, William Jurayj, Xi Wang, Chuanyang Jin, Mehrdad Farajtabar, Eric Nalisnick, and Daniel Khashabi, the research addresses a fundamental bottleneck in long-horizon AI agent operations: the accumulation of stale content that degrades performance and eventually exceeds context limits.

As AI agents tackle increasingly complex, multi-step tasks involving extensive chains of thought and repeated tool calls, their operational logs—known as agent traces—grow rapidly. This accumulation of obsolete information anchors subsequent generations, pulling the model’s focus away from current objectives and eventually outgrowing the available context window. Traditional approaches to mitigating this issue rely on fixed-interval compaction triggered strictly by token thresholds. However, these mechanical triggers pay no attention to the underlying structure of the agent’s reasoning trajectory, running the risk of discarding partial results right in the middle of a critical derivation or an active search phase.

To overcome these limitations, the research team developed SELFCOMPACT, a scaffolding technique that enables the language model itself to determine the optimal moments and methods for context compaction. The framework combines two core inference-time elements working in tandem. The first element is a specialized compaction tool that the model can invoke whenever it needs to summarize its accumulated context. The second element is a lightweight rubric that provides behavioral guidance, specifying precisely when the model should fire the tool—such as when a sub-task has been successfully resolved or when a reasoning trajectory is converging—and when it should suppress compaction, such as during an active derivation or when the agent is stuck.

The authors emphasize that both components are essential for the system to function effectively. On its own, the compaction tool is unevenly utilized across various open-weight models, frequently being invoked at unhelpful moments or ignored entirely. Conversely, a rubric alone lacks the mechanism to execute the compaction action. By pairing the two, SELFCOMPACT elicits effective, adaptive compaction dynamically without requiring any additional model fine-tuning or external supervision.

To validate their approach, the researchers conducted extensive empirical evaluations across six benchmark tasks focusing on competitive mathematics and agentic search, utilizing seven distinct models. The results demonstrate that SELFCOMPACT consistently matches or exceeds the performance of traditional fixed-interval summarization methods while operating at a fraction of the token cost. Compared to a baseline utilizing no summarization at all, SELFCOMPACT improved performance by up to 18.1 points on mathematical reasoning benchmarks and between 5 and 9 points on agentic search tasks, all while achieving a 30 to 70 percent reduction in per-question operational costs.

The findings highlight what the researchers describe as a meta-cognitive gap in current language models. Although unprompted models struggle to reliably recognize when their own context has become degraded or "rotting," supplying a lightweight rubric effectively bridges this gap. This demonstrates that determining when to compact context can be treated as a capability that external scaffolds can successfully supply without requiring expensive model retraining.

Related readings and updates from ongoing research further highlight the broader context of this work in agent development and model alignment. Recent investigations into multi-environment selection, such as the RISED framework, explore training single LLM agents jointly across diverse interactive environments to foster generalist capabilities, addressing challenges in curriculum and data-selection strategies that traditionally overlook relationships between current rollouts across environments. Similarly, separate research into rubric-based alignment addresses the complexities of designing effective reward signals for open-domain question answering, introducing frameworks that generate query-specific rubrics grounded in retrieved evidence to provide fine-grained supervision during complex reasoning tasks.