Self-Hosted LangGraph Agent Evaluation Tool: Putting the Judge on Trial

When Your Evaluation System Rots First

A developer shared an open-source tool called AgentX-Trace-Eval on Reddit, born out of a moment of uncomfortable clarity: their team's entire evaluation pipeline for a LangGraph-based customer service agent had become, in their words, "too embarrassing to look at."

The problems showed up on three levels: a golden test set of roughly 50 cases was abandoned shortly after launch and gradually lost all reference value; the LLM judge was handing out 7s and 8s to nearly every output, making it useless as a differentiator; and regressions were typically discovered not by the eval system, but by customers hitting bugs first.

This is a quiet pain point many AI agent teams know well — evaluation infrastructure left unmaintained and unvalidated, eventually becoming a form of institutional self-deception. The author put it bluntly: he'd rather hear "this thing is useless" than get no feedback at all.

What the Tool Actually Does

Architecturally, the tool takes a minimal self-hosted approach: a single binary, SQLite storage by default. Users simply point a Python SDK or any OpenTelemetry (OTel) exporter at it to start collecting traces.

The core feature chain works like this: any real production trace can be converted into a test case with its provenance attached; LLM-as-judge evaluations run against datasets using the user's own OpenAI / Anthropic / Gemini API keys. The whole thing is Apache 2.0 licensed — no account registration, no billing through the author.

The author considers the most differentiated feature to be scoring the judge itself — comparing the judge's ratings against real-world signals like user downvotes and reported outcomes. This lets you tell whether a metric going wrong means the agent broke, or the judge did. The design targets a blind spot that's long been overlooked in LLM evaluation: when the judge can't be trusted, every downstream metric loses meaning.

The One-Line Bug That Broke an Entire Evaluation

The most contested part of the tool — and the one the author himself is least certain about — is the so-called "eval-fix workflow." It's a Claude Code skill: given an evaluation report, it cross-references each suggested fix against the actual source code, applies only the recommendations that hold up under scrutiny, then reruns the evaluation.

This feature came out of a particularly humbling experience. Their judge kept insisting on "adding a completion check," but the actual bug was a [:400] slice operation inside the evaluation framework's own code — it was truncating every output before the judge ever saw it. The judge had no way of knowing this, because it had never seen the code itself.

The result was dramatic: removing that one line sent scores from 1.86 straight to 10. The episode illustrates a failure mode that's easy to overlook — the problem can live in the eval harness itself, not in the model or agent being evaluated. Having automated workflows cross-check recommendations against source code is specifically designed to catch these "the judge can't see the truth" situations.

Three Honest Questions for the Community

The author didn't frame the post as a product pitch. Instead, he raised a few genuine questions worth sitting with for any team doing agent evaluation:

First: does anyone actually maintain evaluation datasets after launch? Or do test sets rot the same way across the industry — carefully built at project kickoff, then quietly forgotten? This is close to a universal pattern: eval sets are constructed with care early on and then abandoned entirely.

Second: for customer service agents, is LLM-as-judge actually worth it over similarity metrics? This cuts to a real cost-benefit tradeoff in evaluation methodology: LLM judging is more flexible but more expensive and less stable; similarity metrics are cheaper but may be too rigid.

Third: would you trust a closed loop that automatically proposes prompt edits, measures them against a golden set, and asks for your approval? Or does that sound like a footgun waiting to go off? This is a direct challenge to the trend of "AI automatically optimizing AI" — when humans retain final approval, is a semi-automated optimization loop a productivity tool or a liability?

Current Limitations and Positioning

The author is upfront about what's missing: OTel currently only supports HTTP, not gRPC; there are no guardrails features at all. He also fully discloses that he's the developer of the project, with the repo at GitHub: AgentX-ai/AgentX-Trace-Eval.

Looking at the bigger picture, this project's value isn't just in the tool itself — it's in the checklist of questions it raises, reflecting a long-underestimated gap in the production deployment of LLM agents: evaluation systems need to be subjected to the same ongoing scrutiny as the things they evaluate. As more teams push agents into production, "who judges the judge" will become an unavoidable question.

For teams running LangGraph or other framework-based agents, a self-hosted, account-free, open-source evaluation tool at least offers a low-barrier starting point for experimentation. Whether the automated fix loop is actually a good idea — that's probably a question only the community can answer through real-world use.