Pin the Contract, Not the Model: Building a Provider-Swappable Agent Grading System

In an AI Agent bootcamp, a recurring awkward scenario is this: a student completes an assignment using some free model, but two days later when the grader re-runs it, the model has changed, the results have changed, and the grading criteria are effectively meaningless. The root cause isn't the student—it's the lab design. If grading requires calling a live model, then grading itself is non-deterministic.

A recent technical note from a bootcamp practitioner proposed an elegant solution: record once, replay forever, grade offline. The core idea is to "pin the interface contract, not the model." This article draws on that material to unpack the design philosophy, implementation details, and applicable boundaries of this approach.

Why Live Grading Is a Disaster

By the second week of the bootcamp, almost every student hits the same wall: someone builds an Agent loop based on whatever model is free that day, and it runs fine when submitted. Two days later, the grading system re-executes the submission and gets different answers, different tool calls, timeouts, or rate-limit pages.

So what exactly is the grading rubric measuring? Not design ability—but luck at re-run time.

Sure, there's some student responsibility here, but it's mainly a bug in the lab design. The author points out: if the grader must call a live model to determine a score, then the grader is non-deterministic by construction, and no rubric can survive on top of that. This diagnosis hits the nail on the head, shifting the problem from "students rely on unstable resources" to "the grading architecture itself is unreliable."

Four Rules: Nailing Down the Contract

The author now requires every submission to satisfy four rules:

1. Hash the prompt: The same prompt string (ignoring leading/trailing whitespace) maps to the same key.
2. Record responses as JSON cassettes: Plain JSON files, committed to the repository.
3. Grading runs replay only: No network, no API key, no clock dependency.
4. The last live run is for demonstration only: Never the source of truth.

The cleverness of these four rules lies in completely decoupling "model behavior" from "grading logic." The model becomes an environment variable, while grading targets the Agent logic the student designed.

Implementation: Three Files, Fifteen Minutes

The initial setup for the entire lab is very lightweight: create a virtual environment, install httpx and pytest, set up cassettes and lab directories. The core is a Provider protocol—note it's a Protocol, not an abstract base class. Any object with a complete method satisfies the contract.

class Provider(Protocol):
    name: str
    def complete(self, prompt: str) -> str: ...

The prompt_key function strips the prompt, then takes the first 16 characters of its SHA256 hash, ensuring whitespace differences don't produce new records. The Replay class reads cassette files and raises a CassetteMiss exception if no matching hash is found—rather than returning an empty string.

The Recorder class wraps any live Provider, checking the cassette first before calling the external API, avoiding wasted calls when re-running a partial session.

The live adapter (the Live class) is explicitly labeled as a sketch, not verified code—different providers have different endpoints, auth headers, and response structures, so you must consult their respective docs. This honest labeling is commendable, preventing readers from copy-pasting straight into a production incident.

The Provider protocol leverages Python's structural subtyping mechanism: as long as a class implements a complete method and has a name attribute, it automatically satisfies the Provider type without explicit inheritance. Unlike Java-style interfaces, the benefit is that the live adapter, recorder, and replayer can evolve independently, and stub objects can easily replace them in tests. Protocol was introduced in the typing module (Python 3.8+), and when combined with static type checkers (like mypy), contract violations can be caught without running code—further reinforcing the goal of "pinning the interface."

Cassette is a term borrowed from HTTP record-and-replay tools (like VCR.py), originally referring to serializing a network interaction into a replayable file. In this lab, a cassette is plain JSON, with prompt hashes as keys and model response strings as values. Compared to binary or compressed formats, plain-text JSON is easy to review, diff, and version-control, and lets the grader read it without any deserialization dependencies.

Three Checkpoints: Decide in Under a Minute

The author designed three quick checkpoints, each completable in under a minute:

Checkpoint 1: Whitespace is not a new prompt. Two calls differing only in whitespace should hit the same recording.

Checkpoint 2: Misses must be loud. No empty strings, no None, no swallowed exceptions. The CLI must exit non-zero.

Checkpoint 3: Two providers, one score. Feed the same cassette to the grader twice; the output hash must be identical. This is the key check proving the rubric is model-agnostic, and it's where most students fail first—because they hid model_name inside the grading logic.

The first two checkpoints verify the correctness of the infrastructure; the third verifies whether the design philosophy actually landed. Progressive and solidly designed.

The third checkpoint is effectively testing the deterministic scoring function: given the same cassette, the grader's output (including the score and any intermediate hashes) must be byte-for-byte identical. This requires that no non-deterministic sources—timestamps, random numbers, environment variables, or model names—leak into the grading logic. A common student mistake is writing model_name into the grading logic to "distinguish providers," which causes the same cassette to produce different scores under different provider labels—directly violating the "two providers, one score" contract. The correct approach is to let the grader depend only on the prompt-response pairs in the cassette, with provider info recorded as metadata in a trace log, not participating in scoring decisions.

Rubric: Reward Design, Not Luck

The actual rubric the author uses is worth presenting in full:

Criterion	Weight	Full Marks	Typical Failure
Contract stability	30	Same score under two providers	Passes only on the model it happened to use
Loud failure	25	CassetteMiss propagates, exit code non-zero	Returns empty string but looks successful
Tracing	20	Every call logs key, provider, duration	Only logs the final answer
Written summary	15	An honest failure post-mortem	Just writes "it works"
Stretch goal	10	Completes one and records a discovered failure	Started all three but finished none

One detail worth noting: model choice, token consumption, and speed earn zero points. The author explicitly says those are environment variables, not skills. This grading design refocuses students from "tuning models" back to "designing reliable systems."

Boundaries: When This Lab Doesn't Apply

The author proactively lists three no-go zones for the replay approach—and this self-limitation actually increases the credibility of the solution:

• Don't use for tool-call semantics: Different providers serialize function arguments differently, and replay may mask real incompatibilities.
• Don't use for latency or quality benchmarks: That's just evaluating a JSON file.
• Don't use as a production evaluation framework: Production needs random sampling; this is a grading apparatus, not an evaluation platform.

"Boundaries are part of the deliverable. If a student can't articulate what their lab can't measure, their summary isn't finished." This perspective is worth learning from for anyone teaching technical subjects.

A Zero-Cost Lab

For bootcamps with zero budget, the author recommends MonkeyCode's open-source project, which offers free model access and free server options—enough to record cassettes on day one and replay all semester. Disclosure: This article was written as part of a MonkeyCode product promotion. The author also gives an honest warning: free quotas change, so check the console on recording day for the current quota—don't copy numbers from blogs, including this one.

"Recorded cassettes are durable assets; quotas are weather." This metaphor precisely captures the core value of the replay approach: converting unstable external dependencies into stable local assets.

What Students Actually Learn

The ultimate takeaway from this lab is provider-agnostic. The core lesson: anything you can't re-run consistently, you can't grade fairly—including your assumptions about the model itself.

The author suggests creating FAILURES.md in the repo from day one, and says "the second entry is always more interesting than the first." This emphasis on recording failures is precisely the part most easily overlooked in engineering education.

For teams building Agent evaluation systems, this approach offers a valuable middle layer: it doesn't solve the complex problems of production-grade evaluation, but it ensures that teaching and assignment grading aren't held hostage by model randomness. In an era of rapidly churning model providers, the "pin the contract" mindset may outlast any specific tool.

One-line Summary

Bootcamp uses prompt hashing and JSON cassettes for offline replay grading, pinning the interface contract rather than the model.

Paragraph Summary

The article addresses the problem of non-deterministic grading caused by live model calls in AI Agent bootcamps, proposing a "record once, replay forever, grade offline" solution. The core is pinning the Provider interface contract rather than a specific model: SHA256 hash the prompt, store responses as JSON cassettes, run grading on replay only, and use live runs for demonstration only. Three one-minute checkpoints verify the infrastructure and design philosophy, with "two providers, one score" being the key proof of model-agnostic grading. The rubric rewards only contract stability, loud failure, and tracing—giving zero points for model choice, token consumption, and speed. The author also explicitly lists three no-go zones: tool-call semantics, latency/quality benchmarks, and production evaluation frameworks, emphasizing that "recorded cassettes are durable assets; quotas are weather."