SpecCompact: Slot-Parallel Speculation Accelerates LLM Agent Context Compaction

LLM agents generate ever-growing histories during long interactions, making context compaction a key technique for managing this history. However, summarizing long histories with large models introduces significant latency, and the compaction process must balance information retention against response speed. A recent paper proposes the SpecCompact framework, which accelerates compaction through slot-parallel semantic speculation, achieving 2.04–5.63x compaction speedup across multiple benchmarks.

Background: The Latency Dilemma of Compaction

When executing tasks, agents need to compress interaction history into structured summaries so that key information is preserved within a limited context window. Traditional methods such as Hermes-style summarization rely on large models to generate summary content segment by segment. Although quality is high, the serial generation approach causes latency to grow linearly with history length. For agent applications requiring real-time responses, this latency can become a bottleneck.

The authors observed that individual slots in a structured summary can still retain information when generated independently. This means slots are not strictly dependent on one another, opening the door to parallelization. Based on this insight, SpecCompact decomposes the compaction task into multiple slots, uses a small draft model to generate candidate content in parallel, and then has the target large model verify and correct it.

Context compaction is a key mechanism for LLM agents in long-horizon interactions. An agent's context window is limited (commonly 8K to 128K tokens), while multi-turn task execution generates large amounts of history, including tool call results, observations, and reasoning traces. Without compaction, the agent either exceeds the window limit or incurs sharply higher inference costs due to overly long inputs. The goal of compaction is to distill history into a structured summary (with slots such as task goal, completed steps, key findings, and to-dos), preserving necessary information while controlling token count. Unlike traditional text summarization, agent context compaction emphasizes task continuity—the summary must allow the agent to avoid losing state in subsequent steps, so a structured format with fixed slots is often used. This also provides the structural basis for slot-level parallelization.

SpecCompact's Core Mechanism: Slot Parallelism and Semantic Speculation

SpecCompact's workflow has two stages: draft generation and verification/correction. First, a fine-tuned small draft model (drafter) generates candidate lines for each slot in parallel. Then, the target model verifies each line, rejecting unreasonable content and repairing it. The repaired slot prefixes are used for subsequent draft generation, forming an iterative refinement loop.

This design draws on the idea of speculative decoding, but extends the speculation unit from tokens to semantic slots. The draft model is responsible for fast generation, while the target model handles quality control, with a clear division of labor. The paper also introduces system-level optimizations to reduce redundant context processing and model calls, further lowering latency.

Speculative decoding is a technique for accelerating autoregressive generation without changing the target model's output. At its core, a smaller and faster draft model first generates multiple candidate tokens consecutively, and then the target large model verifies these tokens in parallel in a single pass; the accepted prefix is kept directly, and the first rejected position is regenerated by the target model. Because one forward computation of the large model can verify multiple tokens, overall generation speed improves significantly, and the output distribution is theoretically consistent with the target model. SpecCompact migrates this idea from the token level to the semantic slot level: the draft model generates candidate content for each slot in parallel, and the target model verifies and corrects slot by slot. The difference is that the speculation unit is coarser (an entire line of slot content rather than a single token), and the verification criterion shifts from probabilistic acceptance to semantic plausibility. Therefore, it is not guaranteed to be fully equivalent to serial generation, but it gains higher parallelism and a granularity better suited to structured summarization.

Experimental Results: Significant Speedup and Performance Trade-offs

On the AppWorld and τ³ benchmarks, tested with two target models, SpecCompact achieved 2.04–5.63x compaction speedup compared with Hermes-style summarization. In a user study under the Hermes environment, the median compaction speedup reached 8.68x. These data show that slot-parallel speculation can effectively alleviate compaction latency.

However, downstream evaluation also reveals model- and task-dependent performance trade-offs. The speedup is not without cost; on some tasks, the quality of information retention after compaction may decline. The paper candidly presents this trade-off, reminding researchers to weigh it according to task characteristics in practical deployment.

The two benchmarks mentioned each have different emphases. AppWorld is an evaluation environment for interactive code agents, where agents must complete cross-application tasks (such as email, calendar, and payments) through API calls. It emphasizes multi-step planning and state tracking, and its histories are long and highly structured. τ³ (an extended version of tau-bench) focuses on three-way tool-agent-user interaction, simulating real customer service scenarios. It requires agents to follow domain policies and engage in multi-turn dialogue with simulated users, testing the impact of context compaction on policy compliance and user information retention. Both involve long histories and structured state, making them suitable for evaluating compaction quality and latency. Hermes-style summarization refers to the baseline method that generates summaries segment by segment according to predefined slots (such as task, progress, and next step), and is a common context management implementation in agent frameworks.

Analysis: The Significance of Speculative Acceleration in Agent Systems

SpecCompact's value lies in extending the idea of speculative execution from the token level to the semantic level. Traditional speculative decoding requires the draft model and target model to share a vocabulary, whereas slot-level speculation allows coarser-grained parallelism and is better suited to structured summarization tasks. In addition, the method only requires fine-tuning the draft model without changing the target model, giving it good practical feasibility.

From a systems perspective, reducing compaction latency directly improves agent response speed. Especially in multi-turn interaction scenarios, accumulated compaction overhead can consume a large amount of time. By parallelizing and reducing redundant computation, SpecCompact offers a new optimization direction for real-time agent applications.

Limitations and Future Directions

The paper does not discuss in detail the draft model's generalization ability across different domains, nor the impact of slot partitioning strategies on final summary quality. The existence of performance trade-offs means that in some tasks with extremely high requirements for information completeness, more conservative compaction strategies may be needed. Future work could explore adaptive slot partitioning, combine reinforcement learning to optimize the draft model, and extend the method to multimodal history compaction.

Overall, SpecCompact provides a practical acceleration solution for context management in LLM agents, and its slot-parallel speculation approach may also inspire optimization of other structured generation tasks.

One-line Summary

SpecCompact uses slot-parallel semantic speculation to accelerate LLM agent context compaction, achieving 2–5x speedup.

Paragraph Summary

SpecCompact is a framework for accelerating LLM agent context compaction. To address the problem that traditional serial summarization latency grows linearly with history length, it decomposes structured summaries into multiple slots, uses a fine-tuned small draft model to generate candidate content in parallel, and then has the target large model verify and correct each slot, while introducing system-level optimizations to reduce redundant computation. On the AppWorld and τ³ benchmarks, it achieves 2.04–5.63x compaction speedup compared with Hermes-style summarization, with a median speedup of 8.68x in a user study. Downstream evaluation also reveals model- and task-dependent performance trade-offs, namely that speedup may come with a decline in information retention quality. This work extends the idea of speculative execution from the token level to the semantic slot level, providing a practical solution for context management in real-time agents.