AgentX-Trace-Eval
An open-source, self-hosted LLM agent evaluation tool built on a single-binary and SQLite storage architecture. It supports OpenTelemetry trace data collection, enables quality scoring of LLM judges themselves, and is released under the Apache 2.0 license.
Timeline (last 90 days)
AgentX-Trace-Eval is an open-source, self-hosted LangGraph agent evaluation tool that uses a single binary file and SQLite storage by default, and supports collecting trace data via a Python SDK or OpenTelemetry exporter.
AgentX-Trace-Eval is licensed under Apache 2.0, requires no account registration, and does not bill through the author.
The tool's LLM-as-judge uses the user's own OpenAI / Anthropic / Gemini API key to run evaluations against datasets.
The core differentiating design of this tool is scoring the judge itself, comparing the judge's scores against real-world signals (user complaints, actual reported results) to determine whether the problem lies with the Agent or the judge.
The tool includes a Claude Code skill called eval-fix workflow, which cross-references each fix suggestion against the actual source code, applies only the suggestions that hold up to scrutiny, and then re-runs the evaluation.
A bug the author encountered was a [:400] slice operation in the evaluation framework code that truncated every output before the judge could see it; after removing that line, scores shot up from 1.86 to 10.
The tool's OpenTelemetry currently supports only HTTP and does not yet support gRPC, and it has no guardrails functionality at all.
The tool was motivated by the team's experience evaluating a LangGraph-based customer service agent, where a golden test set of approximately 50 cases was abandoned after launch and gradually lost its value.
Any real runtime trace can be converted into a test case along with its provenance information.
All Facts (8)
Any real runtime trace can be converted into a test case along with its provenance information.
50%UnverifiedAgentX-Trace-Eval is licensed under Apache 2.0, requires no account registration, and does not bill through the author.
50%UnverifiedThe tool's LLM-as-judge uses the user's own OpenAI / Anthropic / Gemini API key to run evaluations against datasets.
50%UnverifiedThe core differentiating design of this tool is scoring the judge itself, comparing the judge's scores against real-world signals (user complaints, actual reported results) to determine whether the problem lies with the Agent or the judge.
50%UnverifiedThe tool includes a Claude Code skill called eval-fix workflow, which cross-references each fix suggestion against the actual source code, applies only the suggestions that hold up to scrutiny, and then re-runs the evaluation.
50%UnverifiedAgentX-Trace-Eval is an open-source, self-hosted LangGraph agent evaluation tool that uses a single binary file and SQLite storage by default, and supports collecting trace data via a Python SDK or OpenTelemetry exporter.
50%UnverifiedThe tool was motivated by the team's experience evaluating a LangGraph-based customer service agent, where a golden test set of approximately 50 cases was abandoned after launch and gradually lost its value.
50%UnverifiedA bug the author encountered was a [:400] slice operation in the evaluation framework code that truncated every output before the judge could see it; after removing that line, scores shot up from 1.86 to 10.
50%