AI Agent Gives Different Answers Every Time? Replace Output Matching with Constraint-Based Testing

You spent three days building an AI Agent that rewrites user emails to sound friendlier. Tuesday it was perfect. Wednesday the same input produced something inexplicably formal. Thursday it threw in an unprompted apology and signed off with "Warmly, Your AI." You stare at your test suite — every assertion is red. But the Agent isn't broken. It's just "different today."

This is the most maddening problem in agentic AI: How do you test a system that gives a different answer every time?

Why Traditional Testing Fails Instantly

Anyone who's written unit tests knows the logic: given input X, expect output Y — clean, deterministic, elegant. AI Agents don't play by those rules.

An LLM-based Agent might rephrase the same sentence ten different ways — all correct, all passing a human gut-check, yet none matching your hardcoded expected string. Writing assertEqual(output, "Sure, I'd love to help!") is like testing whether a friend uses the exact same wording every time they say hello.

This isn't a flakiness problem. The testing paradigm itself is fundamentally wrong. There's a natural tension between deterministic assertions and generative outputs, and sticking with the old approach will turn your test suite into a sea of red.

The Counterintuitive Fix: Don't Test Outputs — Test Constraints

The mindset shift that will save your sanity: You don't test what the Agent said. You test what it must never do, what it must always do, and what it must structurally produce.

Here's a great analogy: when testing a chef, you don't check whether the garnish is placed at exactly a 43-degree angle. You check — is the chicken cooked? Is there broken glass in the food? Did they use the ingredients the customer ordered? Agent testing works the same way. Tests should function as behavioral constraints, not output matching.

There are three genuinely effective testing layers that form a practical, deployable methodology.

Layer 1: Structural Contracts

Verify that the output has the right shape. If the Agent returns JSON, validate the schema. If it writes emails, check that subject lines, body text, and required fields are present and non-empty.

import jsonschema

agent_output_schema = {
    "type": "object",
    "required": ["subject", "body", "tone"],
    "properties": {
        "subject": {"type": "string", "minLength": 3},
        "body": {"type": "string", "minLength": 20},
        "tone": {"enum": ["friendly", "professional", "casual"]}
    }
}

def test_email_rewrite_structure(agent, sample_email):
    result = agent.rewrite(sample_email)
    jsonschema.validate(result, agent_output_schema)  # Raises if shape is wrong

Whether the Agent writes "Hey there!" or "Hi friend!", this test passes — because you're testing the contract, not the phrasing.

Layer 2: Guardrail Assertions

These are your "must never" and "must always" rules, catching the issues that actually matter in production:

def test_email_guardrails(agent, sample_email):
    result = agent.rewrite(sample_email)

    # Must never leak the original sender's private data
    assert sample_email["sender_ssn"] not in result["body"]

    # Must always preserve the original recipient
    assert result["to"] == sample_email["to"]

    # Must never exceed token budget (cost control!)
    assert len(result["body"].split()) < 500

    # Tone must match what was requested
    assert result["tone"] == "friendly"

Notice: not a single hardcoded phrase. Every assertion targets a property the output must have, regardless of how the LLM chose to word it. Privacy leaks, runaway costs, tone drift — these are the things that actually break in production.

Layer 3: LLM-as-Judge

For subjective questions — "Is this email actually friendly?" — use a second LLM call to score the output. It sounds circular, but give the judge a rubric and it works surprisingly well:

def test_tone_quality(agent, judge_llm, sample_email):
    result = agent.rewrite(sample_email)

    verdict = judge_llm.evaluate(
        prompt=f"""Rate this rewritten email on friendliness (1-5).
        Original: {sample_email['body']}
        Rewritten: {result['body']}
        Score only. No explanation."""
    )

    score = int(verdict.strip())
    assert score >= 4, f"Friendliness score {score}/5 — too low"

Run it five times. If it passes four times, your Agent is robust enough. If scores oscillate between 2 and 5, your prompt has a problem — and now you have a quantifiable metric instead of a gut feeling.

The reason LLM-as-Judge works is that evaluation and generation are two tasks of different difficulty — much like how human proofreading tends to catch more errors than human writing. The judge LLM doesn't need to create anything; it only needs to make a judgment along one constrained scoring dimension, making its outputs more stable and predictable than generated content. A few practical notes: use a different model or different system prompt for the judge versus the generating Agent, to avoid shared biases that cause them to rubber-stamp each other. Make your rubric as specific as possible — "friendly" should be broken down into actionable sub-criteria like "includes a greeting," "contains a thank-you phrase," and "avoids imperative tone." Finally, sample multiple times and take the median rather than relying on a single judgment to significantly reduce variance. This pattern is already used in academic LLM capability evaluations (e.g., MT-Bench, Chatbot Arena); bringing it into production Agent testing essentially moves mature evaluation engineering practices into your CI/CD pipeline.

The Testing Pyramid for Agents

Apply this structure to every Agent you build:

• Bottom layer (every commit): Structural schema validation. Fast, deterministic, catches output format regressions.
• Middle layer (nightly): Guardrail assertions across 20+ diverse inputs, catching safety/policy violations and behavioral drift.
• Top layer (weekly): LLM-as-Judge evaluation on a curated benchmark set, catching subtle quality degradation.

The bottom layer runs in milliseconds. The top layer costs a few cents per API call. Both are infinitely better than "I tried it once in the playground and it felt fine."

This layered strategy strikes a pragmatic balance between speed, cost, and coverage.

The "testing pyramid" concept here borrows from Mike Cohn's 2009 software testing model: a large base of fast unit tests, a middle tier of integration tests, and a small top tier of slow end-to-end UI tests. The core idea is that lower layers are cheaper while higher layers are closer to real user scenarios — you need to balance cost against coverage. Mapping this to AI Agents: "unit tests" become purely deterministic checks like schema validation, "integration tests" correspond to running guardrail assertions across diverse inputs, and "end-to-end tests" become LLM-as-Judge quality evaluations. Importantly, the traditional pyramid's recommendation to have many bottom-layer tests and few top-layer tests still holds here. Over-relying on LLM-as-Judge makes your test suite slow and expensive, while high coverage at the schema layer is the foundation for fast feedback.

The Thing Nobody Tells You

The core insight is this: the goal of Agent testing isn't to prove your Agent is correct — it's to prove it is safe to deploy. That's a lower bar, and a far more useful one.

Your Agent will surprise you. It will rewrite things in ways you didn't anticipate, sometimes better than you would have. The test suite's job isn't to prevent surprises. It's to prevent disasters.

Here's a practical challenge to close on: take the Agent you're building right now and write three guardrail assertions for it — not output matching, not vibes, but three specific properties it must always satisfy. That's the first step from "debugging by intuition" to engineering with rigor. For any team pushing LLM applications toward production, this shift from output verification to constraint verification may matter more than any framework or tool.