Annotrace
A step-level annotation workbench for LLM agent trajectories, supporting parsing of multiple trajectory formats, human-machine collaborative annotation, and measurement of annotation consistency.
Timeline (last 90 days)
Annotrace supports six trajectory formats, including OpenTelemetry GenAI, LangSmith, and Langfuse.
Annotrace uniformly converts raw trajectory data into an ordered list of nested steps, thereby defining the minimal unit of annotation.
Annotrace allows annotators to write a codebook, frontier models propose an initial annotation set based on trajectory samples provided by annotators, and the models are asked exactly the same questions as human annotators.
Annotrace can measure the agreement between any two annotators, whether human or AI.
On a multi-agent question-answering corpus, Annotrace exactly reproduced the 235 annotation units from prior manual annotation work.
The annotation agreement rate between the two models on the same trajectories reached 91% (κ=0.82).
The open-source nature of Annotrace (inferred from the description) may promote standardization in the agent research field.
All Facts (7)
Annotrace supports six trajectory formats, including OpenTelemetry GenAI, LangSmith, and Langfuse.
50%UnverifiedAnnotrace uniformly converts raw trajectory data into an ordered list of nested steps, thereby defining the minimal unit of annotation.
50%UnverifiedAnnotrace allows annotators to write a codebook, frontier models propose an initial annotation set based on trajectory samples provided by annotators, and the models are asked exactly the same questions as human annotators.
50%UnverifiedAnnotrace can measure the agreement between any two annotators, whether human or AI.
50%UnverifiedOn a multi-agent question-answering corpus, Annotrace exactly reproduced the 235 annotation units from prior manual annotation work.
50%UnverifiedThe annotation agreement rate between the two models on the same trajectories reached 91% (κ=0.82).
50%UnverifiedThe open-source nature of Annotrace (inferred from the description) may promote standardization in the agent research field.
50%