LLM App Getting Worse With Every Change? Build an Effective Evaluation System in Three Steps

Every time you tweak a prompt or switch models, some answers get better while others get worse—a pain point shared by many LLM app developers. On Reddit, one developer posted for help: "Every time we change a prompt or model, something else breaks. One answer gets better, two get worse. We only find out when users complain. Do you run evaluations? When an agent gives a bad answer, how do you trace step by step what it's doing? Right now we can only read logs, and it's too time-consuming."

This post struck a chord. It exposes a core problem: LLM app iteration lacks a reliable feedback loop. This article combines community discussion with engineering practice to break down how to systematically answer "is my app getting better or worse?"

Why "Good" and "Bad" Are Hard to Judge for LLM Apps

Traditional software has clear pass/fail criteria, but LLM outputs are open-ended. One answer getting better may just mean it happened to cover some edge case; two answers getting worse may affect core user experience. What's trickier is that these changes are often only discovered when users complain—by which point the damage is done.

The "reading logs" approach described by the Reddit poster is essentially post-hoc debugging. Logs tell you which steps the agent executed, but they can't tell you "whether this step was correct." When an agent goes off track at step 7 of a 10-step reasoning chain, the log faithfully records the error but doesn't point out where it happened or why.

Multiple developers in the community pointed out that the root cause is the lack of benchmark datasets and automated evaluation. Without a fixed test set, the impact of each change can only be gauged by feel; without automated evaluation, regression testing is impossible.

Building an Evaluation System: From Test Set to Automated Pipeline

Build a test set that represents real scenarios

The first step in evaluation is to collect a representative set of input-output pairs. These samples should cover core functionality paths, common edge cases, and known historical failure cases. Some in the Reddit discussion suggested sampling from production logs, prioritizing cases with poor user feedback and high-frequency scenarios.

The test set doesn't need to be large—50-100 high-quality samples can expose most regression issues initially. The key is that each sample must have a clear expected output or scoring rubric—whether that's exact match, key information inclusion, or scoring by another LLM according to a rubric (LLM-as-judge).

Run regression tests before every change

With a test set in place, you can automatically run evaluations after each prompt or model change, comparing the pass rates of old and new versions. This is similar to unit tests in traditional software, but evaluates semantic correctness.

One developer shared their approach: divide the test set into "must-pass" core cases and "best-effort" edge cases, with any core case failure blocking release. This tiered strategy maintains a quality baseline while avoiding excessive constraints on iteration speed.

Scale evaluation with LLM-as-judge

Manual annotation is costly, and LLM-as-judge can automate scoring at larger scale. The approach is to use a more capable model (such as GPT-4 level) to score outputs according to a detailed rubric. Community experience shows that the more specific the judge's rubric (e.g., "does the answer contain three key facts"), the more stable the scoring.

But beware of the judge's own biases: it may favor verbose or formatted answers. Therefore, judge scores should be periodically aligned with manual spot checks.

Common Biases in LLM-as-judge and How to Mitigate Them

Beyond favoring verbosity and formatted output, judge models also exhibit position bias (tending to choose the answer that appears first), self-preference (scoring their own generated answers higher), and authority bias (more readily endorsing content with a confident tone). In practice, this can be mitigated by: first, randomly swapping the order of answers being compared; second, using a model from a different family than the generation model as the judge; third, explicitly requiring the judge to list its reasoning before assigning a score in the rubric; and fourth, cross-validating the same batch of samples with two different judges, escalating to manual review when their scores diverge significantly.

Choosing Evaluation Metrics

Regression testing metrics shouldn't be limited to a single pass rate. It's recommended to also track: exact match rate (for scenarios with definite answers), key fact recall rate (whether the answer covers required points), and refusal rate and hallucination rate (whether the answer fabricates non-existent information). Looking at multiple metrics together helps avoid hidden degradation like "pass rate goes up but answers become more conservative/verbose."

Tracing Agent Behavior: Step-Level Visualization Beyond Logs

The poster's second question is equally critical: when an agent gives a bad answer, how do you trace it step by step?

The problem with pure logs is information overload and lack of context. A better approach is structured tracing: recording each step's input, output, tool calls, and state changes, presented as a visual timeline. This lets you directly pinpoint "which decision at which step caused the error."

Open-source tools like LangSmith and Phoenix provide this kind of tracing capability. Their core value isn't recording more logs, but organizing execution chains into a queryable, comparable structure. For example, you can filter for all cases where the "retrieval" step timed out, or compare step differences between two versions given the same input.

Going further, you can link tracing data with evaluation results: when a test case fails, jump directly to the corresponding execution trace to see which step deviated from expectations. This "evaluation-tracing" integration can dramatically shorten debugging time.

OpenTelemetry and the Standardization of LLM Tracing

The underlying philosophy of structured tracing aligns with distributed system observability: treat an agent run as a trace, with each reasoning step and tool call as a span recording start/end times, inputs/outputs, and metadata. In recent years, OpenTelemetry's GenAI semantic conventions have been driving standardization of LLM tracing fields, enabling traces from different frameworks (LangChain, LlamaIndex, custom agents) to be collected and queried with the same backend, avoiding vendor lock-in. Another benefit of standardization is the ability to integrate LLM calls with existing APM systems, seeing the correlation between "increased model latency" and "downstream service timeouts" in the same view.

Building a Continuous Feedback Loop

Evaluation and tracing aren't one-time efforts. A healthy LLM app needs to connect user feedback, production monitoring, and automated evaluation into a closed loop:

• Production side: Collect explicit user feedback (thumbs up/down) and implicit signals (retries, abandonments, edits).
• Evaluation side: Regularly add production failure cases to the test set, keeping it in sync with the real distribution.
• Tracing side: Automatically save complete execution traces for failure cases for root cause analysis.

A consensus that repeatedly emerged in the Reddit discussion: don't wait for users to complain before acting. Make evaluation a fixed part of the release process and tracing the default debugging tool, so that every iteration truly moves toward "getting better."

Back to the original question—how do you know whether your app is getting better or worse? The answer isn't gut feeling, but a repeatable, measurable, traceable engineering system. It requires upfront investment, but what you get in return is both iteration speed and quality.

One-line Summary

Build a feedback loop for LLM apps with test sets, automated evaluation, and structured tracing—instead of relying on user complaints to discover degradation.

Paragraph Summary

Starting from a Reddit developer's help request, the article identifies the core pain point of LLM app iteration: the lack of a reliable feedback loop. Open-ended outputs make "good or bad" hard to judge, and reading logs is post-hoc debugging that can't pinpoint wrong decisions. The solution has three layers: first, build a test set covering core paths, edge cases, and historical failures, run regression tests before every prompt or model change, and set release red lines for core cases; second, scale evaluation with LLM-as-judge while aligning with manual spot checks to offset its biases; third, use tools like LangSmith and Phoenix for structured step-level tracing, linking traces with evaluation results to quickly locate which step went off track. Ultimately, production feedback, automated evaluation, and execution tracing must be connected into a continuous loop for measurable iteration.