OnTrack
A streaming monitoring method for real-time supervision of LLM agent trajectories. Using optimal transport, it performs structure-aware comparison between the dependency graph of the agent's current step and a reference trajectory, enabling millisecond-level anomaly detection and decisions to warn or block without an additional LLM.
Timeline (last 90 days)
OnTrack is a streaming monitoring method that, by performing a structure-aware comparison of the Agent's current step and its dependency relationships with recorded successful run trajectories, without additionally calling an LLM as a judge, judges in real time and decides to remind the user or block the Agent.
OnTrack uses Optimal Transport to measure the structural difference between the current trajectory and the reference trajectory, rather than a simple string or embedding similarity comparison.
OnTrack can complete the structural comparison judgment for each step of the Agent in about 1 millisecond.
In SWE-bench trajectory evaluation, when using information from the first 8 steps, OnTrack's ability to rank failed trajectories is better than content similarity methods, with an AUROC higher by 0.057.
OnTrack's abort strategy saved about 18% of the computational resources that would otherwise have been consumed on failed runs, and among the interrupted runs 83% were indeed heading toward failure.
OnTrack's capability decreases as data access privileges decrease: with full reference access (having historical trajectories and tool schemas) it can perform plan violation detection; with medium access (only schemas) it can only discover abnormal calls by checking against the legal usage of tools; with no prior knowledge (only real-time logs) it can still identify loops, stagnation, and repeated tool calls, but cannot judge whether an unfamiliar but reasonable plan deviates from the goal.
OnTrack does not require additional training of a guard model and mainly relies on existing run data.
OnTrack's limitation is that the quality of the reference trajectories directly determines the monitoring ceiling; if the historical successful runs themselves have problems, the comparison baseline is unreliable.
OnTrack's millisecond-level monitoring detects the deviation of a single step from the reference structure; slow drift across multiple steps requires combining a longer history window to judge.
All Facts (9)
OnTrack is a streaming monitoring method that, by performing a structure-aware comparison of the Agent's current step and its dependency relationships with recorded successful run trajectories, without additionally calling an LLM as a judge, judges in real time and decides to remind the user or block the Agent.
50%UnverifiedOnTrack uses Optimal Transport to measure the structural difference between the current trajectory and the reference trajectory, rather than a simple string or embedding similarity comparison.
50%UnverifiedOnTrack can complete the structural comparison judgment for each step of the Agent in about 1 millisecond.
50%UnverifiedIn SWE-bench trajectory evaluation, when using information from the first 8 steps, OnTrack's ability to rank failed trajectories is better than content similarity methods, with an AUROC higher by 0.057.
50%UnverifiedOnTrack's abort strategy saved about 18% of the computational resources that would otherwise have been consumed on failed runs, and among the interrupted runs 83% were indeed heading toward failure.
50%UnverifiedOnTrack's capability decreases as data access privileges decrease: with full reference access (having historical trajectories and tool schemas) it can perform plan violation detection; with medium access (only schemas) it can only discover abnormal calls by checking against the legal usage of tools; with no prior knowledge (only real-time logs) it can still identify loops, stagnation, and repeated tool calls, but cannot judge whether an unfamiliar but reasonable plan deviates from the goal.
50%UnverifiedOnTrack does not require additional training of a guard model and mainly relies on existing run data.
50%UnverifiedOnTrack's limitation is that the quality of the reference trajectories directly determines the monitoring ceiling; if the historical successful runs themselves have problems, the comparison baseline is unreliable.
50%UnverifiedOnTrack's millisecond-level monitoring detects the deviation of a single step from the reference structure; slow drift across multiple steps requires combining a longer history window to judge.
50%