TraceSpec: A Speculative Decoding Optimization Framework for LLM Agents

Speculative decoding is a common technique for accelerating large language model (LLM) inference, but traditional evaluation metrics often aggregate multiple decisions within agent execution into a single statistic, making it difficult to reveal where draft supervision truly needs strengthening. The recently proposed TraceSpec framework offers a new optimization approach for speculative decoding in LLM agents by connecting trajectory-level evaluation with adaptive draft training.

The Blind Spot of Aggregate Metrics: As Context Grows Longer, Local Acceptance Rates Actually Decline

LLM agents execute tasks over a history of trajectories composed of reasoning, actions, and tool feedback by repeatedly invoking the model. Existing speculative decoding evaluations typically report aggregate acceptance, but this merged statistic obscures differences across decision points. The TraceSpec team built the TraceSpec-Traj dataset, containing 10,000 training states and 2,500 test states, covering two task categories: browsing and software engineering. Analysis of five speculative decoding baselines revealed a counterintuitive phenomenon: aggregate acceptance may rise with context length, yet in matched late-stage browsing actions, acceptance rates actually decline.

This finding indicates that relying solely on overall metrics can misdirect draft model training. Workload type and task stage further distinguish these trends, showing that supervision signals should be guided by trajectory conditions rather than distributed uniformly. For example, in the later stages of browsing tasks, the model faces longer histories and more complex decisions, requiring stronger targeted training for the draft model.

Speculative Decoding and Acceptance Rate: Speculative decoding is an inference acceleration technique whose core idea is to use a lightweight draft model to rapidly generate multiple candidate tokens, which are then verified in parallel by the target large model. If the draft model's predicted tokens match the target model's output, they are "accepted"; otherwise they are rejected and rolled back. Acceptance rate refers to the proportion of draft tokens accepted by the target model and is a key metric for measuring draft quality; acceptance length refers to the number of consecutively accepted tokens in each verification round. Traditional evaluation often reports aggregate acceptance rate, i.e., statistics merged across all decision points, but in agent tasks the context and decision complexity vary greatly across turns, so aggregate metrics may mask local performance fluctuations. TraceSpec's trajectory-level evaluation reveals that in late-stage browsing actions, despite longer contexts, local acceptance rates decline—indicating that the draft model performs worse on long-history, complex decision points and requires targeted supervision signals.

How TraceSpec Works: Dynamic Execution and Fixed-Prefix Replay

The core of TraceSpec is the combination of dynamic execution and fixed-prefix replay. Dynamic execution simulates the agent's real interaction process, while fixed-prefix replay allows evaluating the draft model's performance across different trajectory states under controlled conditions. Based on this evaluation, TraceSpec trains on intermediate assistant turns and allocates a fixed prediction budget using state-conditioned, prefix-aware acceptance gap estimation.

Simply put, this method no longer treats all turns equally but dynamically determines where to focus draft supervision based on the current trajectory state. This adaptive allocation strategy aims to improve the draft model's acceptance rate at critical decision points, thereby accelerating the overall inference process.

Fixed-prefix replay is an evaluation technique: given a prefix in the agent's interaction history (i.e., the complete trajectory up to a certain turn), the prefix is held fixed, and the draft model generates subsequent tokens multiple times under identical conditions, with the target model's acceptance recorded. This method eliminates noise from randomness in dynamic interaction, making comparisons between different draft models or training strategies fairer and more reproducible. State-conditioned, prefix-aware acceptance gap estimation refers to estimating the acceptance gap between the draft model and the target model based on the current trajectory state (such as task type, turn, and context length), and dynamically allocating the prediction budget accordingly (i.e., how many candidate tokens the draft model generates at once). Budget allocation is not uniform but skewed toward states with large acceptance gaps and high supervision needs, thereby maximizing overall acceleration under limited computational resources.

Experimental Results: Throughput Gains and the Value of Adaptive Allocation

Across three target model sizes and two temperature settings, TraceSpec improved replay throughput by 12.5% to 22.6% compared to its DSpark baseline. Ablation experiments on Qwen3-4B further showed that adaptive allocation improved acceptance length by 3.3% to 10.0% compared to fixed all-turn supervision. Additional comparative experiments confirmed the roles of state conditioning and consecutive-token agreement.

These results demonstrate that trajectory-aware supervision strategies can effectively improve draft model quality. Interestingly, the throughput gains vary with model size and temperature, indicating that the method adapts to different deployment conditions.

Consecutive-token agreement is a fine-grained metric for evaluating draft model quality, measuring the extent to which multiple consecutive tokens generated by the draft model exactly match the target model's output. Unlike single-token acceptance rate, it focuses on the coherence of the draft sequence: even if individual tokens are accepted, frequent rejection of subsequent tokens still limits overall acceleration. High consecutive-token agreement means the draft model can more stably predict the target model's output sequence, thereby reducing the overhead of verification rollbacks. In TraceSpec's ablation experiments, this metric was used to verify the improvement in draft model coherence from state conditioning and adaptive budget allocation, further corroborating the effectiveness of trajectory-aware supervision.

Implications for Accelerating Agent Inference

TraceSpec's contribution lies not only in performance gains but also in providing a set of evaluation resources and trajectory findings that connect agent execution analysis with practical draft improvements. For LLM agent applications that require frequent model invocations, this trajectory-conditioned speculative decoding optimization is expected to reduce inference latency and improve response speed.

As agents become widely adopted in scenarios such as browsing and programming, inference optimization tailored to their execution characteristics will become increasingly important. TraceSpec's approach—shifting from aggregate metrics to trajectory-aware fine-grained supervision—offers a direction that future research can build upon.

One-line Summary

TraceSpec improves speculative decoding efficiency for LLM agents through trajectory-level evaluation and adaptive draft training.

Paragraph Summary

The TraceSpec framework addresses the problem that traditional aggregate metrics in LLM agent speculative decoding mask local performance differences, proposing a trajectory-aware evaluation and training method. The study built the TraceSpec-Traj dataset with 10,000 training states and 2,500 test states, covering browsing and software engineering tasks, and found that aggregate acceptance may rise with longer contexts while local acceptance in matched late-stage browsing actions declines. TraceSpec combines dynamic execution with fixed-prefix replay, trains on intermediate assistant turns, and allocates prediction budgets based on state-conditioned, prefix-aware acceptance gap estimation. Experiments show that compared to the DSpark baseline, replay throughput improves by 12.5% to 22.6%; Qwen3-4B ablation experiments show adaptive allocation improves acceptance length by 3.3% to 10.0%. This work offers a new direction for accelerating agent inference—shifting from aggregate metrics to fine-grained trajectory supervision.