Forge: Adding Reliability Guardrails to Local Model Tool Calling

The Real Pain Point of Building Agents with Local Models

When building agents with local models, the most frustrating problem usually isn't that the model "can't call tools at all" — it's that it's "almost right." It might wrap JSON in a code block, misspell a function name, skip a required prerequisite step, or return an answer before the tool has even finished running. These seemingly minor errors are enough to break an entire task chain midway through.

Forge (as in "to forge") is an open-source project built specifically to address this problem — a Python reliability middleware layer that sits between the LLM and the tool execution layer. With roughly 2.2k stars, it targets developers using Ollama, llama.cpp, vLLM, or the Anthropic API. It intercepts the model's tool calls: fixing what can be fixed, and guiding the model to retry what can't.

Why "Almost Right" Gets Amplified Over Multiple Steps

Looking at any single step, the model might only be slightly wrong. But multi-step tasks stack these small errors on top of each other. The author provides a straightforward example: if each step has a 90% success rate, the probability of 5 consecutive steps all succeeding is only about 59%.

This is the core problem Forge aims to solve. It doesn't try to improve the model's business reasoning — it focuses on the mechanical reliability of tool calling: turning format errors, unknown tools, and step violations into problems that can be detected, reported back, and retried a limited number of times.

Three-Layer Guardrail Mechanism

Forge's core is a three-layer progressive protection system that covers the complete loop from format validation to error correction and retry.

Layer 1: Response Validation

Every time a model is about to call a tool, Forge checks whether the tool name exists in the actual tool list provided by the current request, whether the call structure is well-formed, and whether the parameters are valid JSON objects. Note that deeper parameter schema and business-logic validation remains the responsibility of the client or tool execution layer — Forge only enforces correctness at the protocol level.

Layer 2: Rescue Parsing

Some models express the correct intent but use non-standard formatting — wrapping JSON in code blocks, using Mistral-style tool calls, or Qwen-style XML tags. Forge attempts to extract a valid tool call from these non-standard outputs. If it can be rescued directly, there's no need to waste an extra inference round-trip.

Layer 3: Error-Correcting Retry

For outputs that can't be rescued, Forge converts them into explicit error feedback and prompts the model to resample, with a default maximum of three retries. It can also optionally inject a synthetic respond tool, which helps smaller models stay on the more stable tool-calling track even when they should be returning plain text.

Two Integration Paths

Forge offers two distinct usage modes for different needs.

Path 1: Proxy Mode (Lowest Migration Cost)

For developers with existing clients, proxy mode requires the least refactoring. After installing forge-guardrails, you place Forge between your client and the model server: the model backend keeps running on port 8080, Forge listens on another port (e.g., 8081), and the client simply redirects its API address to Forge.

The proxy is compatible with both OpenAI Chat Completions and Anthropic Messages APIs, so tools like OpenCode, Continue, Aider, and Cline can all connect to it — and Claude Code can connect via the Anthropic address. The client thinks it's talking to a more reliable model; in reality, Forge is handling validation, rescue, and retries in the middle.

The boundaries of proxy mode are worth understanding: it works on a per-request basis, does not execute the client's tools, and is not responsible for required steps, prerequisite dependencies, cross-request memory, or workflow state. Even if the client requests streaming, Forge waits for inference to complete before converting to SSE — it's not true token-by-token streaming.

Path 2: Workflow Runner (Full Control of the Agent Loop)

If you want control over the complete agent loop, the Workflow Runner is the right choice. You define tools, parameters, backends, and context budgets in Python, and declare which steps must be completed, which tools depend on which prerequisites, and which termination tool signals that the task is truly done.

For example, generating a research report could require "search first, then read sources, only then allow report generation." If the model tries to skip the search and finish early, the step guardrail will intercept it and inform it which steps are still missing. The Workflow Runner also handles tool execution, context compression, and the full lifecycle of the agent loop.

Forge also includes a Slot Worker that manages shared inference slots with a priority queue, suitable for multiple specialized workflows sharing a single GPU. The project is explicit that this is not multi-agent orchestration — Forge doesn't do multi-agent graphs, DAG planning, or cross-agent coordination. It guards reliability within a single agent loop.

Benchmark Data: How Significant Is the Improvement?

The repository includes a 26-scenario evaluation suite covering multi-step calls, state transitions, context compression, and harder reasoning tasks. The author cites a directly reproducible comparison: the Mistral 3.8B instruction model with Q4 quantization and native llama.cpp calling showed an overall score improvement from 27.3% to 78.3% — the same calling pattern, the only difference being whether Forge guardrails were present.

This demonstrates that mechanical errors can be significantly reduced. However, the author is careful to point out: these results only represent specific configurations within the project's own benchmark suite and do not imply an equivalent improvement in the model's general capabilities. Results are affected by model generation, quantization method, backend, and whether native function calling or prompt injection is used. The safest approach is to run regression tests against your own tools and failure cases.

Getting Started and Known Limitations

Requirements

Forge requires Python 3.12 or higher and does not come with a model — you still need a running inference backend. The official recommendation is llama.cpp Server, as the top-performing configurations typically come from this path; Ollama is easier to install; vLLM is better for high-throughput scenarios; llamafile works for single-file deployments; and Anthropic can serve as a remote baseline reference.

When using a local backend, inference data stays on your machine. However, if you connect to a remote API, you can no longer claim to be "fully offline."

Caution with Multimodal Scenarios

The current version warrants special attention for multimodal use cases: Forge's internal message content is primarily text-based. Image and other multi-part content may be preserved only during the initial proxy pass-through; once conversion, retry, or compression paths are triggered, non-text blocks may be lost. For vision tasks, you must fully validate with your own requests and cannot assume stable support by default.

Who Is Forge For?

The author clearly defines the boundaries: Forge is not a code generator that writes complete applications for you; not a multi-agent orchestrator that coordinates a team of agents; and not a model management tool with a UI.

It's best suited for three types of developers:

• Those whose local small-model tool calls frequently "almost succeed";
• Those who already have an OpenAI- or Anthropic-compatible client and just want to quickly add guardrails via proxy;
• Those building domain-specific agents who need step constraints, context management, and reproducible evaluations.

If you're just doing casual chat, or you truly need complex multi-agent planning, Forge doesn't cover that. The project is currently in Beta on PyPI — run your own failure cases and regression tests before deploying to production.

Summary

Forge's greatest value is bridging the gap between a model "roughly knowing what to do" and "tool calls having a better chance of executing according to protocol." Start with proxy mode if you want minimal code changes; use Workflow Runner for full loop control; or embed just the Guardrails middleware if you already have an orchestrator.

It won't turn an 8B model into a frontier model, but it will eliminate a lot of format errors and wasted retries that never should have happened. For local agent development, this kind of "mechanical reliability" is often more practical than piling on another flashy feature.