Free Model Endpoints: Why You Need a Contract Test Bench, Not Leaderboard Screenshots

Title

Free Model Endpoints: Why You Need a Contract Test Bench, Not Leaderboard Screenshots

Meta Description

Free model endpoints often show JSON format drift, timeouts, and token accounting anomalies. This article explains how to replace leaderboard screenshots with a contract test bench—using fixed fixtures, controlled variables, and failure-class counts—to judge whether an endpoint can become a stable dependency.

Article Content

Wire a small internal tool to a free model endpoint. The first call returns clean JSON. The second wraps the same object in a Markdown code block. The third times out after 30 seconds. The fourth reports 900 completion tokens for a 40-token answer. The demo still looks fine, but your CI job is already broken.

That gap is exactly why free endpoints need a "contract test bench." You are not asking whether the model is the smartest—you are asking whether the endpoint can repeatedly return the data shape your code needs, within the time and token budget you can tolerate. Those are two different questions, and only the second protects a build.

MonkeyCode is an open-source project whose current promotional materials advertise free model access, a free server option, and 10 million free tokens. All three should be treated as time-limited commercial terms rather than technical guarantees—verify current offers before relying on them. This article was prepared as part of a MonkeyCode product promotion.

In this workflow, free access is better used as a probe target than as proof of quality. Even if you switch to another endpoint, the test bench below remains useful.

Write the Contract Before the Prompt

Leaderboards ask which model wins. A contract test bench asks which promises survive repetition. Start from the promises your code already makes.

Contract	Probe	Pass Evidence	Failure Class
JSON object	Request an object with two keys	Whole response parses without stripping code blocks	format_drift
Schema	Fixed key set and values	Keys match exactly	schema_miss
Truncation	Small max_tokens with a long request	finish_reason present and sensible	truncation_ambiguity
Token accounting	Known short prompt	0 < completion_tokens <= max_tokens	accounting_anomaly
Timeout	Client-enforced timeout	Request ends within budget	timeout
Error contract	Malformed request	Non-2xx with structured body	error_contract
Streaming (if used)	Streaming response	Final event arrives	stream_hang

This table is deliberately narrow. It does not test reasoning, factuality, or code quality—those need a separate dataset. It tests whether this endpoint is, for one client, a stable dependency.

Contract testing originates in microservices and API integration. It verifies whether a "provider" satisfies the interface promises a "consumer" depends on, without caring about the service's internal implementation. Traditional contract testing often uses tools like Pact to compare request/response patterns between services; in the LLM context, the promises being verified extend from HTTP status codes and field types to output format, token accounting, truncation semantics, and stream termination—model-specific behaviors. Unlike unit testing, contract testing deliberately does not chase coverage; it locks down a few shape constraints that are critical to the caller, which makes it suitable as a fast gate in CI.

finish_reason is a field returned by OpenAI-compatible chat-completions APIs that explains why generation stopped. Common values include stop (hit a stop sequence or ended naturally), length (truncated at the max_tokens limit), content_filter (blocked by safety policy), and tool_calls (the model requested a tool call). It is the key evidence for judging the "truncation" contract: if finish_reason is length and the client still treats the content as a complete answer, that is the truncation_ambiguity described above.

Build a Small, Frozen Probe Set

Start with four cases and expand to at least 20 before making a team decision. Shape matters more than quantity.

Use four groups of cases: strict JSON extraction (with and without code-block instructions); truncation stress (answers that cannot fit in the token budget); Unicode and whitespace (transport layers normalize text in surprising ways); and malformed-request controls (sent separately from the main run).

Store cases as JSONL with id, group, prompt, and expect fields. Hash the file and record the hash in your notes. If the fixture changes, the old numbers belong to a different experiment. Do not put keys, customer text, or production prompts in fixtures. Free services may log requests—treat every probe as public.

JSONL (JSON Lines) is a text format with one independent JSON object per line, convenient for streaming reads, appends, and line-by-line diffs. In a test bench, each case is one line with fields id, group, prompt, and expect, so fixtures can be hashed, versioned, and reviewed case by case. Compared with a single large JSON array, JSONL is easier to debug on partial writes or merge conflicts, and it keeps results separate from fixtures.

Run with Controls

Controls are what separate a test bench from a screenshot.

1. Set temperature to 0 where the endpoint accepts it; if ignored, record that fact.
2. Fix max_tokens, the client timeout, and the client version for the whole run.
3. Run three attempts per case. Measure inside without retries; count retries as new attempts.
4. Repeat in two windows at least two hours apart. Free capacity is not stationary.
5. Shuffle case order with a fixed seed so one slow request does not poison every later sample.
6. Record status, latency, usage, finish reason, and a hash of the raw response body—never the API key.
7. Send one malformed request. A 200 is a contract failure, not a "quirky success."

A p95 latency computed from 12 attempts is a range check, not an SLA estimate. That is fine—you are looking for failure classes and severe instability, not precise capacity numbers.

An Adaptable Test Harness

The template below uses only the Python standard library. It is an unexecuted template: it assumes a chat-completions-shaped payload and a BASE_URL. When switching APIs, replace call_endpoint and keep the record shape and scoring logic.

import hashlib
import json
import os
import random
import time
import urllib.error
import urllib.request
from pathlib import Path

BASE_URL = os.environ['BASE_URL'].rstrip('/')
API_KEY = os.environ['API_KEY']
MODEL = os.environ.get('MODEL', 'default')
TIMEOUT_S = 25.0
MAX_TOKENS = 64

CASES = [
    {'id': 'json-01', 'group': 'strict_json',
     'prompt': 'Return only a JSON object with keys ok and count. count must be 2.',
     'kind': 'json'},
    {'id': 'json-02', 'group': 'strict_json',
     'prompt': 'Return only a JSON object with keys ok and count. count must be 2. Do not use a code fence.',
     'kind': 'json'},
    {'id': 'long-01', 'group': 'truncation',
     'prompt': 'Write a 1000-word explanation of HTTP caching.', 'kind': 'text'},
    {'id': 'unicode-01', 'group': 'unicode',
     'prompt': 'Return the exact text: café / 第二行 / end', 'kind': 'text'},
]

# call_endpoint / score / main match the original; full implementation omitted here

Save it as bench.py and run it with BASE_URL='https://your-endpoint.example' API_KEY='...' python bench.py. Run the malformed-request control separately: change the payload to invalid JSON, assert the status is 4xx and the body is parseable, and do not merge that result into the pass count.

Read Counts, Not Composite Scores

A single score hides the one failure you actually need to fix. Report a set of separate numbers:

Metric	How to Report	Why It Matters
Strict JSON	passed / attempts per group	Shows format drift directly
Timeouts	count / attempts	Exposes client vs. capacity failures
Token anomalies	count	Catches impossible usage accounting
Latency	p50 and p95, plus max	Shows budget risk without hiding outliers
Error contract	valid 4xx / malformed requests	Shows whether failures are machine-readable
Raw evidence	Fixture hash and response body hashes	Lets another person reproduce the run

Use counts when denominators are small. "9/12 strict JSON passed, 3 format_drift" is useful. "75% quality" is not. The latter invites a comparison the test bench cannot support.

p50 and p95 latency are percentile metrics: p50 (median) means half the requests were faster than that value, and p95 means 95% of requests were faster. Compared with the mean, percentiles better expose long tails—a few 30-second timeouts will pull the mean up without masking a worsening p95. The original emphasizes that a p95 computed from 12 attempts is only a range check, because p95 itself is unstable at such small sample sizes; it is good for finding "severe instability" but not for use as a capacity SLA.

Why These Numbers Are Not Marketing

Marketing numbers compress time, sample size, and failure detail into one score, one date, and one model label. They are easy to read and easy to misuse. A contract test bench reports fixture hash, client version, denominators, failure histogram, and time window. Those details are boring, but they are exactly what lets another developer reproduce or disprove your result.

Do not turn the test bench into a ranking. If you test two endpoints, keep fixtures and clients fixed, report each failure class separately, and state how many attempts you made. If the free offer changes, your old numbers are a historical observation, not a current capability claim. That is the difference between evidence and a screenshot: a screenshot shows one good moment; a test bench shows the conditions under which that moment occurred.

Decision Table

Workload	Minimum Evidence	When Not to Use a Free Endpoint
One-off prototype	12 attempts, malformed control, classified failures	The data cannot be sent under current terms
CI job	30+ attempts across two windows, p95 below job timeout, fallback path	Timeouts would block every merge
Internal tool	Contract pass counts plus error-budget monitoring	You need hard latency commitments
Production dependency	Contract test bench plus fallback, monitoring, and data policy	You need an SLA the free offer does not provide

This table is a gate, not a guarantee. Passing means the endpoint is worth retesting—not that the model is correct.

Limitations and Who Should Skip This

This test bench does not measure reasoning quality, factuality, or code correctness, and it will not tell you which model writes better patches—that requires a separate evaluation. The samples are small: 12 or 30 attempts can reveal severe instability but cannot prove steady-state capacity. Free offers are also time-limited; 10 million tokens and a free server option are commercial terms, not contractual guarantees, and may change without notice.

Unless current terms explicitly allow it, do not send keys, regulated data, or customer content to free endpoints. Do not treat this workflow as your only production control. Teams that need deterministic latency, strict data residency, or contractual SLAs should not use free endpoints as production dependencies.

Where the Free Server Fits

Treat the free server option as a disposable probe runner: fixtures, result JSONL, and summaries live in storage you control. If the server resets, you should lose one run, not your evidence. Inject API keys at runtime—never into fixtures, the repository, or result logs. Treat the runner as cattle, not as a database.

The useful output is not a slogan about a model, but a sentence like this: under fixture hash abc123 and client version 0.4.1, 7 of 12 strict JSON attempts passed across two windows, and 5 failed with format drift. That sentence survives a demo, a handoff, and a skeptical reviewer.

If you want a low-risk starting point, the project's current free-access and free-server offers are enough to run once. Verify the current terms, keep the raw counts, and let the test bench decide the next step.

One-line Summary

Use a contract test bench instead of a leaderboard to verify whether a free model endpoint can reliably return the data shape your code needs.

Paragraph Summary

The article points out that free model endpoints exhibit unstable behavior across repeated calls—format drift, timeouts, token accounting anomalies—and leaderboards cannot answer whether "this endpoint can repeatedly return the shape my code needs." The author proposes a "contract test bench" method: first define seven categories of promises (JSON, schema, truncation, token accounting, timeout, error contract, and streaming), then run them using a small frozen JSONL probe set, fixed temperature and max_tokens, three attempts per case, two time windows, and fixed-seed shuffling, and finally report counts by failure class rather than a composite score. The article provides an adaptable template based on the Python standard library and a decision table, and stresses that free offers are commercial terms rather than technical guarantees—free endpoints should not serve as production dependencies that require an SLA.