Building an Agent Harness by Hand Without Frameworks: A Four-Hour Deep Dive

When an agent framework breaks, the most frustrating part usually isn't the error itself—it's that you have no idea which layer is doing the retry, which tool's return value blew up the context, or which guardrail rule got bypassed. A developer on Reddit nailed this pain point in a single sentence: frameworks are great to use, until you can't understand their behavior.

The "Black Box" Cost of Frameworks

Agent frameworks wrap tool calls, context management, retry logic, and safety guardrails into a few lines of configuration. That convenience comes with a clear cost.

When you see a tool call triggered three times, it could be the model retrying on its own, the framework's retry policy, or your tool registry doing idempotency handling—three layers are all suspects, but the logs can't tell them apart. Context overflow is the same story: is it the raw JSON returned by a tool being too large, a summarization strategy that didn't kick in, or a misconfigured memory tier?

This ambiguity is especially dangerous in production. When an indirect prompt injection makes the agent execute SQL it shouldn't, you can dig through the framework docs forever and still not find which layer's guardrail failed.

The author's take is blunt: write a harness yourself in plain Python once, and afterward, any framework source code you read, you'll immediately see which layer made which trade-off.

Four Core Modules a Hand-Written Harness Must Address

The core of this workshop is building an AI data analyst that doesn't depend on any agent framework. The four modules map directly to the areas most likely to break in production.

Tool Registry: Validate Before Executing

A tool registry isn't just maintaining a list of functions. It needs to dynamically discover tools from real MCP servers, validate parameters before invocation, and enforce permission checks.

The order matters: permissions before validation. Many frameworks conflate these two steps, causing a call that should be rejected to first execute parameter parsing, burning tokens and time.

MCP (Model Context Protocol) is an open protocol proposed by Anthropic that lets models communicate with external tools and data sources in a standardized way. It normalizes tool definitions, parameter schemas, and call results into a unified format, enabling interoperability across different models and tool servers. "Dynamically discovering tools from real MCP servers" means the agent requests the tool list from the MCP server at startup, rather than hardcoding tools in code. The trade-off: parameter validation and permission checks must happen at runtime, because you don't know in advance which tools you'll get. This is exactly why the tool registry must put "validation" and "permissions" before execution—once tools come from dynamic discovery, any unvalidated call could trigger unintended external actions.

Context Manager: Work Within a Hard Token Budget

Context isn't about "cramming in as much as possible"—it has a hard budget. This module needs to implement summarization, expandable handles, and tiered memory.

Expandable handles are a clever design: large tool return values are stored as handles first, and only expanded when the specific content is needed, avoiding filling up context all at once. Tiered memory distinguishes short-term conversation from long-term knowledge, each with different eviction strategies.

Indirect prompt injection differs from ordinary user-input injection: the malicious instruction isn't typed directly by the user, but hidden in external content the agent will read—web pages, PDFs, database fields, JSON returned by tools. When that content gets stuffed into context, the model may treat it as an instruction to execute. Traditional defenses filter user input, but that line of defense is completely ineffective against indirect injection, because the attack payload travels through the "data" channel. The effective interception point is checking tool results before they enter context, identifying whether they contain content disguised as instructions. This is also why the context manager and the guardrail pipeline must work together: the context manager decides what content gets in, the guardrails decide whether what gets in is safe.

Guardrail Pipeline: Layered Interception

A single guardrail is easy to bypass, so you need layers. This pipeline must cover two typical threats: indirect prompt injection and unsafe SQL.

Indirect injection means malicious content is hidden inside data returned by tools, rather than in the prompt the user directly inputs. Traditional input filtering is useless against this; you must check before tool results enter context.

Run Supervisor: Timeouts, Retries, Replanning, Budgets, and Traces

This is the layer most easily hidden by frameworks. The run supervisor handles timeouts, retry policies, replanning after failure, budget control, and structured traces.

Structured traces are the key to troubleshooting. When the agent behaves abnormally, you can see the basis for every decision from the trace, rather than guessing.

Structured traces refer to recording every step of an agent run in a machine-readable structure: the input at each step, decisions, tool calls, return values, latency, tokens consumed, and the reason that triggered that decision. The difference from ordinary logs is that logs are streams of text for humans to read, while traces are event streams with a schema that programs can query, aggregate, and replay. When a tool call is retried three times, the trace can tell you clearly which layer the retry happened at and what condition triggered it. Without traces, troubleshooting relies on print statements and guesswork; with traces, you can compare normal and abnormal runs and pinpoint which layer's policy caused the deviation.

Why This Exercise Is Worth Doing Once

The author emphasizes that the exercise can be done on a laptop with a mock model and a local database—no GPU or API key required. This setting lowers the barrier and is closer to real development scenarios—most agent logic issues have nothing to do with model capability, but rather with harness design.

The workshop is led by Bruno Gonçalves, founder of Data For Science, and runs four hours. But the real value isn't in those four hours—it's in the clarity you gain when you revisit your existing stack with the perspective of "I've written a harness."

Frameworks won't disappear, nor should they. But at minimum, you should know what decisions they're making on your behalf.

One-line Summary

Building a plain-Python agent harness by hand reveals the retry, context, and guardrail decisions frameworks hide.

Paragraph Summary

The article points out that agent frameworks wrap tool calls, context management, retry logic, and safety guardrails into a few lines of configuration, but the cost of that convenience is opaque behavior—when a tool call fires three times, you can't tell whether it's the model, the framework, or the tool registry retrying; context overflow and bypassed guardrails are equally hard to trace to a layer. The author advocates writing a plain-Python AI data analyst harness without framework dependencies, focusing on four modules: a tool registry (dynamic MCP tool discovery, validate-then-execute, permissions first), a context manager (hard token budget, summarization, expandable handles, tiered memory), a layered guardrail pipeline (intercepting indirect prompt injection before tool results enter context), and a run supervisor (timeouts, retries, replanning, budgets, and structured traces). The exercise can be completed with a mock model and a local database—no GPU or API key needed—and the real value lies in gaining a clear perspective when re-reading any framework's source code.