Should an Agent's SQL Drafting Loop Run on Hosted Free Models or a Free Server You Control?

An agent trying to draft a report query made forty-one attempts, and no single one of them looked slow on its own. The real cost only becomes visible when you plot the number of attempts against wall-clock time and the prompt-token count per retry on the same chart. What actually matters is that the deployment location of the drafting step — not the model quality itself — determines this volatility. That observation kicked off a recurring debate with a colleague: should an agent's SQL drafting loop run on a shared hosted endpoint, or on a free server you control yourself?

Why deployment location matters more than model quality

The task itself is mundane: the agent reads a schema summary, drafts SQL, executes it read-only, reads the error, and revises. Every revision resends a growing conversation history, so the twelfth attempt carries far more prompt tokens than the first.

When the drafting step runs on a shared hosted endpoint, retry latency and quota pressure are someone else's scheduling problem; when it runs on a machine you control, latency becomes your problem, but the shape of the retry storm is yours to decide. The author admits upfront there is no single benchmark that settles this for everyone, so what's offered instead is a reproducible harness, the four signals it collects, and a decision rule a team can argue about using its own data.

According to the original piece, MonkeyCode advertises free model access and a free server option, which maps neatly onto the fork in this discussion. The author discloses that this article is part of an outreach effort for that product, describes themselves as a free-tier user rather than a benchmarking authority, and treats the free token allowance (the operator claims ten million tokens on signup) and the free server as claims current at the time of writing, recommending readers verify on the project page themselves.

What are prompt tokens and context bloat?

In an LLM call, prompt tokens refer to the number of tokens in the input sent to the model, including the system prompt, schema summary, conversation history, and the current instruction; completion tokens are the output the model generates. The cost of an agent's SQL drafting loop climbs over time precisely because each revision resends the entire prior conversation history as-is — the longer the history, the more prompt tokens, and the higher the cost of each retry. This is what's called "context bloat": once the history accumulates over dozens of turns, the instructions that actually matter get diluted as a share of total tokens, which drives up both latency and cost and can also reduce the model's attention to the most recent error. Understanding the prompt-to-completion ratio is the key to telling whether a loop is being dragged down by history resending or by generation itself.

Position A: Borrow the hosted free model tier

The first position is that an agent's drafting loop should default to the hosted free tier, because startup cost is measured in minutes rather than days. Authenticate, point the loop at the endpoint, and the schema summary, prompt assembly, and retry logic all stay where they are. For a single developer validating whether an agent can produce usable queries, this is the shortest path to evidence.

The second argument is operational: nobody needs to patch, restart, or monitor anything. If the endpoint degrades, the retry logic you already wrote absorbs it, and you can rerun the same task set tomorrow.

The counterargument (which the author leans toward) is that a shared free tier gives you no control over retry limits, so a runaway loop stays invisible until it's already stalled.

Position B: Self-host the drafting step on a free server

The second position is that the drafting step belongs on a machine you control, even if that machine was free. Once the model runs adjacent to the database, the schema summary never crosses your boundary, and concurrency, retries, and context growth all get capped by you. For regulated schemas or loops that touch production metadata, that boundary is often the entire argument.

Self-hosting also makes failure modes legible. When a retry storm starts, what you see is memory, queue depth, and per-request latency on your own dashboard, not numbers on someone else's status page. The cost is real too: someone has to own the image, the model weights, the restart policy, and the capacity ceiling.

A deployment-location harness you can actually run

What the author offers is a proposed harness, not a validated benchmark. It writes one JSONL line per drafting attempt, so the deployment decision rests on your data rather than the author's opinion. The token counter is just a cheap proxy and the backend is a dry-run echo, so it doesn't depend on any fictional vendor numbers.

# bench_placement.py — proposed harness; the author has not yet run it against MonkeyCode.
import argparse, hashlib, json, statistics, time
from dataclasses import asdict, dataclass

@dataclass
class Attempt:
    task_id: str
    placement: str      # "hosted-free" | "free-server"
    attempt_no: int
    prompt_tokens: int
    completion_tokens: int
    wall_ms: int
    ok: bool
    error_class: str | None
    sql_sha256: str | None

Steps to run the comparison

1. Write twenty tasks into tasks.txt, mixing simple queries with two or three joins, and share the same file across both deployment locations.
2. Wire your backend into draft_sql and keep the return contract stable, so the JSONL schema doesn't change between the two runs.
3. Run python bench_placement.py --placements hosted-free --out hosted.jsonl first, then repeat with --placements free-server.
4. Compare attempts per task and p95 wall-clock time rather than average latency, because retry storms hide in the tail.
5. Regardless of where it's deployed, gate every generated statement before execution, using a read-only transaction with a timeout.

-- Cheap guardrail: never let a draft touch data before a read-only plan.
BEGIN;
SET LOCAL transaction_read_only = on;
SET LOCAL statement_timeout = '3s';
EXPLAIN (FORMAT JSON) /* generated statement goes here */;
ROLLBACK;

The four signals that actually decide the direction

Four signals in the JSONL file carry most of the decision weight. Attempts per task tell you how much you're paying to resend context; the prompt-to-completion token ratio tells you whether the loop is dominated by conversation history or by generation; tail latency tells you how badly the shared tier is handling your retry pattern; and the schema's sensitivity level tells you whether the boundary argument outweighs all the numbers above it.

Signal	Hosted free tier wins when	Free server wins when
Attempts per task	Roughly under four	Over eight, and you can cap concurrency
Schema sensitivity	Public, fixture, or synthetic data only	Internal or regulated metadata
Tail latency (p95)	You can tolerate queueing	You need a stable ceiling you control
Ops capacity	Nobody can take over a machine	Someone owns restarts and capacity

Why look at p95 instead of average latency?

A retry storm is characterized by most requests returning quickly, while a few take extremely long because of queueing, rate limiting, or oversized context. The average gets diluted by all the "normal" requests and masks the tail problem; p95 (the latency ceiling experienced by 95% of requests) directly exposes the slowest slice. For an agent drafting loop, one tail timeout often triggers the next retry, which stuffs more history into the following request and amplifies subsequent latency, forming a positive feedback loop. That's why, when evaluating deployment location, the combination of p95 and attempt count reflects real experience far better than average latency, and comes closer to the kind of "occasionally freezes" complaint users actually raise.

Failure analysis for each side

The most common hosted-side failure is a silent quota exhaustion mid-loop: the agent keeps retrying, the endpoint keeps rejecting, and the loop reports it as a schema error. You have to instrument error classes separately, or you'll misread rate limiting as a bad join. The second is context bloat: by the fifteenth attempt, the history carried is so large that the real instruction becomes a rounding error.

Server-side failures are usually resource exhaustion rather than rejection. One hot model plus three concurrent agents can exceed memory, and the process dies without any useful information. Cold starts also distort early latency, so drop the first task of each run, or you're comparing startup cost against steady state. Neither deployment location fixes a wrong query, and that's precisely the failure people most want deployment location to solve.

A decision rule worth defending

Start every new agent loop on the hosted free tier and get the harness running, because evidence beats architecture debates. Once attempts per task cross eight, or the schema becomes sensitive, or tail latency becomes the constraint users complain about, move the drafting step to a free server. Either way, put the harness in version control, because the same numbers that justified the move will also justify moving back.

Decisions like this are only as good as the alternatives you've actually tried. If you want to test both deployment locations on the same task set, MonkeyCode's free model access and free server option give you two configurations you can run immediately, without a procurement process.

Limitations and who should skip this topic

This harness measures deployment location, not correctness; the token counter is a proxy rather than a real tokenizer, so treat the numbers as relative. The free token allowance and free server are operator-claimed availability that the author did not independently benchmark; self-hosting on a free server still requires someone to own the image.

If you need deterministic sub-second p99 latency, or compliance rules forbid third-party inference, the decision is already made before any benchmark runs, and you should skip this debate entirely. Also skip it if your agent has never produced a query a reviewer approved — deployment location can't save a loop that hasn't yet proven it works.