OnTrack: Real-Time LLM Agent Trajectory Monitoring with Optimal Transport

LLM Agents are expanding from travel planning and stock trading into high-stakes scenarios like IT incident triage. The problem is that most agents operate almost fully autonomously, with woefully few rule-based safeguards. Once an irreversible action is executed, both costs and safety risks materialize simultaneously. The two mainstream approaches to date each have weaknesses: one introduces a guard agent to supervise step by step, at the cost of added overhead and latency at every step; the other performs post-hoc log evaluation, but by the time it reaches a conclusion, tokens have already been burned and damage has already occurred.

OnTrack's Core Idea: Comparing Agent Trajectories as Structures

This paper proposes OnTrack, which takes a third path: streaming monitoring. Instead of summoning an additional LLM to act as judge, it performs structure-aware comparison between the agent's current step and its dependency relationships against recorded successful execution trajectories, completing each step's judgment in about 1 millisecond, then deciding whether to alert the user or directly block the agent.

The keyword here is "structure-aware." An agent's step is not isolated text—it forms a dependency graph with prior tool calls, parameters, and returned results. OnTrack uses Optimal Transport (OT) to measure structural differences between the current trajectory and reference trajectories, rather than simply comparing strings or embedding similarity. This explains why it can catch anomalies like loops, stalls, and repeated tool calls—cases that "look normal in content but have already gone off track structurally."

Optimal Transport (OT) is a mathematical framework for measuring the difference between two probability distributions, originally proposed by Monge in the 18th century to solve the minimum-cost plan for moving a pile of earth to another construction site, hence the name "Earth Mover's Distance." Given two distributions and a cost matrix between them, OT finds the matching that minimizes total transport cost, from which the Wasserstein distance can be defined.

In the trajectory monitoring context, the node distribution of the current step's dependency graph and the node distribution of the reference trajectory can be treated as two probability distributions, with structural differences between nodes as the cost. OT then yields a structural distance that accounts for global matching. Compared to element-by-element string or embedding comparison, OT allows "misaligned but overall similar" structures to be correctly matched, making it more robust to parameter order changes and equivalent tool substitutions, while remaining sensitive to topological anomalies like loops, stalls, and repeated calls.

The cost is that it requires solving a linear programming problem, but on single-step dependency graphs with small node counts, approximation algorithms like Sinkhorn iteration can compress the time to milliseconds—this is precisely the foundation for OnTrack's roughly 1-millisecond per-step judgment.

Capability Boundaries Across Three Data Access Scenarios

The paper breaks the problem into three scenarios with decreasing access privileges, and the conclusions are quite honest—the less data, the weaker the monitoring capability.

Full reference access: Both historical successful trajectories and tool schemas are available. Here OnTrack can perform plan violation detection, determining whether the agent has deviated from known viable execution paths.

Medium access: Only tool schemas, no historical trajectories. Monitoring degrades to detecting anomalous calls by checking against the tool's own legal usage.

No prior knowledge: Only real-time step logs. Here the system can still identify loops, stalls, and repeated tool calls, but cannot determine whether an unfamiliar yet reasonable plan has deviated from the goal.

This layered design has practical significance: many teams haven't accumulated enough high-quality successful trajectories but still want a basic real-time line of defense. OnTrack's capability expectations decrease as data access decreases, but it doesn't pretend to be omnipotent when data is scarce.

Measured Results on SWE-bench

The paper evaluated on SWE-bench trajectories, using information from the first 8 steps. The result is superior ranking ability for failed trajectories compared to content similarity methods, with AUROC higher by 0.057. This number isn't stunning, but distinguishing success from failure trajectories in early steps is inherently difficult.

A more practical benefit comes from the abort strategy: OnTrack saves about 18% of the computational resources that would otherwise be consumed on failed runs. And among interrupted runs, 83% were indeed heading toward failure—5 out of 6 aborts were correct judgments. This means it doesn't trade massive false positives for savings; precision is in a usable range.

AUROC (Area Under the Receiver Operating Characteristic curve) is a common metric for evaluating the ranking ability of binary classification models, ranging from 0.5 (random) to 1.0 (perfect). It equals the probability that, given a randomly chosen positive sample and a randomly chosen negative sample, the model scores the positive sample higher. It essentially measures "whether failed trajectories can be ranked ahead of successful ones," rather than accuracy at a fixed threshold. In monitoring scenarios, AUROC is more appropriate than accuracy because the final abort threshold can be adjusted separately based on business tolerance for false positives.

SWE-bench is a benchmark for evaluating LLM capability to fix issues in real GitHub repositories, containing thousands of bug-fix tasks from real Python projects, each equipped with runnable tests to verify whether the patch passes. Agent trajectories on this benchmark typically contain dozens of steps of file reading, editing, and test invocations, making it ideal material for studying long-horizon agent failure patterns. The "first 8 steps of information" mentioned in the text refers to making judgments using only the structure of the trajectory's first 8 steps, which is closer to the actual deployment scenario where a decision on whether to interrupt must be made early.

Why Streaming Monitoring Deserves More Attention Than Guard Agents

The problem with the guard agent approach isn't just cost. Calling an additional LLM at every step introduces latency jitter, which is a poor experience in interactive scenarios; moreover, the guard agent itself can be bypassed or make mistakes, creating a new attack surface. Post-hoc log evaluation, meanwhile, cannot prevent irreversible operations at all—it can only be used for retrospective analysis.

OnTrack compresses monitoring cost to the millisecond level and doesn't rely on additional model inference, making it possible to embed as standard middleware in agent runtimes. Of course, its limitations are also obvious: the quality of reference trajectories directly determines the monitoring ceiling. If historical successful runs themselves have problems, the comparison baseline is unreliable.

Implications for Agent Engineering Practice

For teams currently deploying agents, OnTrack suggests a pragmatic direction: rather than pursuing an omniscient, omnipotent guard model, first record trajectory structures and use lightweight comparison as the first line of defense. Historical data from successful runs is itself an asset, worth being structurally preserved rather than discarded as mere logs.

Another point worth noting is the design of the abort strategy. The 18% computational savings and 83% abort accuracy mean that about 17% of aborts are still false positives. Which scenarios can tolerate false positives and which scenarios would rather let things pass than interrupt—this is a strategic question requiring business trade-offs, and the paper itself doesn't provide a universal answer.

The trade-off between false positives and false negatives is a core design issue for anomaly detection systems. A false positive occurs when the system wrongly aborts a run that would have succeeded, at the cost of wasted computation already invested and potentially missing the correct answer; a false negative occurs when the system lets a run heading toward failure continue, at the cost of continued resource consumption or even executing irreversible operations.

In agent scenarios, this trade-off is more subtle than traditional monitoring: irreversible operations (such as transfers, data deletion, sending emails) cannot be rolled back once executed, so such steps should lean toward "prefer false positives"; reversible steps like read-only queries and draft generation are suitable for relaxed thresholds to avoid interrupting reasonable exploration paths.

A practical engineering approach is to tag each action with a "reversibility level" based on tool type, and set independent abort thresholds for different levels, rather than using a single score for the entire trajectory. Additionally, false positive rates will vary with the quality and coverage of the reference trajectory library; after deployment, teams should continuously collect runs that were falsely aborted but would have actually succeeded, feeding them back into the trajectory library to form a closed loop.

Frequently Asked Questions

Does OnTrack require training a model? No additional guard model training is needed—it performs trajectory structure comparison based on optimal transport, relying mainly on existing run data.

Can it be used in non-SWE-bench scenarios? The paper validated on SWE-bench, but the method itself targets general agent trajectories, provided that steps and dependency structures can be obtained.

Will millisecond-level monitoring miss slow anomalies? It detects deviation between single steps and reference structures; slow drift across multiple steps requires combining longer historical windows for judgment.

One-line Summary

OnTrack monitors agent trajectories with millisecond-level structural comparison, saving 18% of failed computation while achieving 83% abort accuracy.

Paragraph Summary

OnTrack proposes a streaming monitoring approach that uses optimal transport to compare the dependency structure of an agent's current step against historical successful trajectories, determining deviation in about 1 millisecond. This avoids the extra inference cost and latency jitter of guard agents, and addresses the inability of post-hoc log evaluation to prevent irreversible operations. The paper distinguishes three data access scenarios—full reference, medium, and no prior knowledge—with monitoring capability decreasing as data decreases. On the first 8 steps of SWE-bench trajectories, its AUROC for ranking failed trajectories is 0.057 higher than content similarity methods; the abort strategy saves about 18% of failed run computation, with 83% of abort decisions correct. The limitation is that reference trajectory quality determines the monitoring ceiling, and the paper doesn't provide a universal trade-off scheme for false positive tolerance.