OpenVLHarness: How Multimodal Memory Reshapes Tool Use in Visual Agents

The Bottleneck of Visual Reasoning: Internal Thinking Is No Longer Enough

Scores on visual reasoning benchmarks for multimodal large models climb year after year, but an awkward reality remains: many tasks simply cannot be solved by "thinking" alone. The OpenVLHarness team ran a controlled experiment covering 22 benchmarks, and the conclusion was direct — simply increasing the base model's inference budget yields quite limited gains.

The reason is easy to understand. Real-world visual tasks often require three kinds of external evidence: solid perceptual results, external knowledge, and numerical measurements obtained through heterogeneous tools. This evidence must be converted into observations the model can read and continuously retained across multi-step reasoning. Most current tool-calling approaches only solve the "calling" link, without addressing how evidence is remembered and reused.

Inference budget in the context of multimodal large models usually refers to the number of tokens or computational steps the model can use for "internal thinking" before generating a final answer — for example, increasing chain-of-thought length, allowing more self-sampling, or extending reasoning time. OpenVLHarness's controlled experiment fixed the tools and data and adjusted only this budget, finding that gains saturate quickly. This shows that much of the key information in visual tasks does not exist in the model parameters or input pixels, but must be obtained through external perception, knowledge retrieval, or numerical computation; simply extending the internal reasoning chain only makes the model repeatedly self-confirm in the absence of evidence, and can even amplify hallucinations. Therefore, inference budget should be treated as a limited resource whose marginal benefit depends on whether external evidence can be effectively introduced and reused.

The OpenVLHarness Architecture: Capability Layer + Persistent Multimodal Memory

OpenVLHarness positions itself as a "multimodal reasoning harness," with two core components.

Capability Layer: Unifying Heterogeneous Tools and Rendering Outputs into Readable Evidence

The capability layer combines task-level tool interfaces with evidence rendering. Its value lies in smoothing over differences among backend tools, providing consistent access, and converting tool-returned results (images, structured data, web references, etc.) into observation formats the model can read. This step may seem like an engineering detail, but it actually determines whether the model can truly "understand" tool output rather than being disrupted by raw return formats.

Evidence rendering refers to converting the raw outputs returned by heterogeneous tools (such as JSON, HTML, image tensors, database records) into observation formats that multimodal models can directly consume, for example normalized images, annotated table screenshots, or condensed structured text. The return schemas of different backend tools vary greatly; if directly concatenated into the prompt, the model needs extra reasoning to parse the format, which both consumes budget and is error-prone. Evidence rendering lowers the threshold for the model to "understand" tool output by unifying schemas, trimming redundant fields, and preserving key visual elements. It determines whether tool calls can truly be transformed into effective evidence, rather than merely formal call success.

Persistent Multimodal Memory: Making Evidence Addressable Artifacts

Persistent multimodal memory is the key that distinguishes this approach from ordinary tool-calling frameworks. It retains the corresponding images, structured data, and web references throughout the entire query process, storing them as addressable artifacts. The model can revisit this evidence at any time in subsequent reasoning steps, rather than relying on the temporary memory of the context window.

In addition, a coding interface allows the model to execute generated programs on stored data, thereby completing computations that require combining multiple tool results. Together, these three mechanisms support the iterative process of "acquire—inspect—combine" multimodal evidence.

Addressable artifacts refer to structured objects that are assigned unique identifiers during reasoning and can be retrieved and referenced on demand. Unlike temporary tokens in the context window, artifacts exist independently of the conversation history, and the model can revisit original images, tables, or web snippets at any subsequent step through identifiers. This design alleviates two common problems: first, evidence in long contexts being diluted or truncated by subsequent tokens; second, the same evidence needing to be transmitted repeatedly when called multiple times, driving up token costs. Artifactization also enables the coding interface to directly execute programs on stored objects, such as performing aggregate computations on multiple charts, without requiring the model to restate intermediate results in text. Its essence is converting "memory" from a context management problem into a storage and indexing problem.

Experimental Data: Average Gains of 9.7–12.6 Points, at Lower Cost

In evaluations across 22 benchmarks and eight open-source and proprietary backbone models, OpenVLHarness improved on matched no-tool baselines by an average of 9.7–12.6 points. Specifically: Qwen3-VL-8B improved by 11.2 points, Kimi K3 by 9.7 points, and GPT-6 Astra by 9.9 points.

More noteworthy is the efficiency dimension. Running OpenVLHarness with GPT-6 Luna at medium reasoning effort outperforms the base model, Codex, and Visual Sketchpad at the highest reasoning effort, leading by 7.7–8.5 points, while model token cost is at most 37% of the latter. This shows that how external evidence is organized can exchange for more solid reasoning quality without piling on compute.

Why This Work Is Worth Watching

Most existing visual agent research focuses on expanding tool types or optimizing calling strategies. OpenVLHarness asks a more fundamental question: how can tool outputs be stably understood by the model, and how can evidence avoid being lost across multi-step reasoning. Persistent multimodal memory answers by turning evidence into addressable artifacts — a different technical route from simply enlarging the context window.

For teams building real-world visual applications (such as document understanding, chart analysis, and cross-source information verification), the thinking behind this harness offers an engineering paradigm worth borrowing: first solve the readability and reusability of evidence, then discuss the number and complexity of tools.

One-line Summary

OpenVLHarness organizes external evidence with persistent multimodal memory, improving visual reasoning by an average of 9.7–12.6 points at lower cost.

Paragraph Summary

Scores of multimodal large models on visual reasoning benchmarks are climbing, but the OpenVLHarness team's controlled experiment covering 22 benchmarks shows that simply increasing the base model's inference budget yields limited gains, because real visual tasks require three kinds of external evidence — perceptual results, external knowledge, and numerical measurements from heterogeneous tools — and this evidence must be converted into readable observations and continuously retained across multi-step reasoning. OpenVLHarness consists of a capability layer (unified tool interfaces and evidence rendering) and persistent multimodal memory (storing images, structured data, and web references as addressable artifacts), and supports a coding interface for executing programs on stored data. It improves by an average of 9.7–12.6 points across eight backbone models; with GPT-6 Luna at medium reasoning effort, it leads high-reasoning-effort baselines by 7.7–8.5 points at at most 37% of the token cost.