Annotrace: A Step-Level Annotation Workbench for Agent Trajectories

Title

Annotrace: A Step-Level Annotation Workbench for Agent Trajectories

Meta Description

Annotrace is a workbench for step-level annotation of agent trajectories, supporting six trajectory formats, introducing human-AI collaborative annotation and consistency measurement. It reproduces 235 annotation units on a multi-agent QA corpus, with 91% inter-model agreement.

Article Content

Researching why LLM agents fail often requires manually annotating trajectories, and every study ends up reinventing the wheel. While observability platforms can record agent trajectories, they do not provide structured annotation; traditional annotation tools can only handle text and do not understand steps, delegation, or tool calls, leaving the minimal unit of annotation ambiguous. Annotrace is an annotation workbench born to solve this pain point. It supports six trajectory formats, transforms each run into an ordered list of nested steps, and introduces frontier models to assist annotation, making human-AI collaborative annotation possible.

Why step-level annotation?

Agent behavior consists of a series of steps, such as planning, tool calls, delegation to sub-agents, and so on. To analyze failure causes, these steps must be annotated at a fine grain. However, existing observability platforms (such as LangSmith and Langfuse) only record and do not provide annotation functionality; generic text annotation tools lack an understanding of agent execution structure and cannot define the basic unit of a "step." As a result, researchers have to build annotation tools from scratch every time, which is both inefficient and hard to reuse. Annotrace exists precisely to fill this gap.

Observability originates from control theory and distributed systems monitoring, referring to the ability to infer a system's internal state from external signals such as logs, metrics, and traces. In the LLM agent context, observability platforms (such as LangSmith and Langfuse) record agent execution in units of traces, where each trace consists of several spans; spans can be nested and carry metadata such as tool call parameters and model inputs and outputs. OpenTelemetry GenAI is the semantic convention of the OpenTelemetry specification extended for generative AI, defining field naming for events such as model calls and tool usage so that traces produced by different frameworks can interoperate. The core goal of observability platforms is debugging and monitoring, so they focus on complete recording and visualization. They do not provide annotation data structures oriented toward research analysis, nor do they abstract traces into a unified step unit that can be judged by humans or models. This is exactly why Annotrace needs to add another layer of transformation on top of them.

Core features of Annotrace

Annotrace's design revolves around three key capabilities: multi-format trajectory parsing, human-AI collaborative annotation, and annotation consistency measurement.

Multi-format trajectory parsing

Annotrace can read six common trajectory formats, including OpenTelemetry GenAI, LangSmith, and Langfuse. It uniformly converts the raw data of these formats into an ordered list of nested steps, thereby clearly defining the minimal unit of annotation. In this way, researchers do not need to worry about underlying format differences and can operate directly at the step level.

Human-AI collaborative annotation

Annotators can write their own annotation codebook, and frontier models will propose an initial annotation set based on the trajectory samples provided by the annotator. The model first answers the question and is asked exactly the same questions as human annotators, turning the annotation process into a joint human-AI action. This design preserves the domain knowledge of human experts while leveraging the efficiency and consistency of AI.

A codebook is a standard tool in social science content analysis and machine learning data annotation. It refers to a document that clearly lists each label's definition, judgment criteria, positive and negative examples, and boundary cases. Its role is to externalize implicit annotation intuition into reusable rules so that multiple annotators can make consistent judgments when facing the same trajectory. In agent trajectory annotation, a codebook typically needs to specify what counts as a step, how calls delegated to sub-agents are scored, and whether tool call failures form a separate category. Annotrace has annotators first write a codebook and then lets frontier models generate an initial annotation set accordingly. In essence, this treats the codebook as a shared instruction for both humans and models, which is also the prerequisite for the tool to achieve human-model comparable annotation.

Annotation consistency measurement

Annotrace can measure the consistency between any two annotators (whether human or AI). On a multi-agent question-answering corpus, Annotrace exactly reproduced 235 annotation units from prior manual annotation work, and the annotation agreement between two models on the same trajectories reached 91% (κ=0.82). This demonstrates the tool's reliability in annotation quality.

Cohen's κ is a statistic that measures agreement between two annotators while correcting for chance agreement. Its value usually ranges from 0 to 1, where 0 means no better than random guessing and 1 means complete agreement. Compared with a simple agreement rate, κ avoids the illusion of inflated agreement when a certain label appears at a very high frequency. Empirical thresholds commonly used in research are roughly: κ above 0.8 is considered high agreement, and 0.6 to 0.8 is substantial agreement. The reported agreement rate of 91% and κ=0.82 indicate that the two models gave the same label on the vast majority of annotation units, and that this agreement was not caused by skewed label distributions, so it can serve as quantitative evidence of the tool's annotation quality and reliability.

Practical results and significance

Annotrace's performance on the multi-agent question-answering corpus is impressive: it not only reproduces the results of manual annotation but also demonstrates the high consistency of AI annotation. This means researchers can use Annotrace to quickly launch annotation projects, reduce repetitive work, and improve annotation efficiency and quality through human-AI collaboration. In addition, the tool's open-source nature (inferred from the description) may promote standardization in the agent research field.

Future outlook

Although Annotrace solves the basic problem of step-level annotation, the complexity of agent behavior continues to grow. In the future, the tool may need to support more trajectory formats, more flexible annotation modes, and deeper automated analysis. Nevertheless, as foundational work, Annotrace provides powerful infrastructure for agent failure research and is expected to drive the entire field forward.

One-line Summary

Annotrace is a step-level annotation workbench for LLM agents that supports multi-format trajectory parsing and human-AI collaborative annotation.

Paragraph Summary

Annotrace is an annotation workbench for LLM agent failure research, designed to fill the gap between observability platforms that only record but do not annotate and generic text annotation tools that do not understand agent execution structure. It supports six trajectory formats including OpenTelemetry GenAI, LangSmith, and Langfuse, and uniformly transforms each run into an ordered list of nested steps, thereby clarifying the minimal unit of annotation. In the annotation workflow, annotators write a codebook, and frontier models propose an initial annotation set accordingly and answer exactly the same questions as human annotators, forming joint human-AI annotation. The tool can also measure consistency between any two annotators. On a multi-agent question-answering corpus, Annotrace reproduced 235 annotation units from prior manual annotation, with 91% agreement between two models (κ=0.82), validating its reliability.