No Labels Needed: Marvo Uses Margin Reward for Self-Evolving Visual Agents

Visual agents are becoming an important paradigm for multimodal tasks—they can alternate between reasoning and tool calls to handle complex tasks that language models alone struggle with. But training such agents typically requires large amounts of human-annotated question-answer pairs or expert demonstrations, which is extremely costly. A new study proposes Marvo, a method for self-evolution from unlabeled images that lets agents learn to call tools effectively without any human supervision.

Dual-Agent Architecture: Challenger and Solver

The core design of Marvo is a pair of agents that compete with each other. The first is the Challenger, which automatically generates question-answer pairs from unlabeled images; the second is the Solver, which answers these questions through reinforcement learning and, in the process, learns when and how to call tools.

The cleverness of this design lies in the division of labor: the Challenger is responsible for producing "learning material," while the Solver is responsible for absorbing capabilities from it. The entire process requires no human-annotated question-answer data—only a batch of raw images.

Margin Reward: What Kind of Questions Are Most Worth Practicing?

Self-evolving training faces a key challenge: if the Challenger generates questions that are too easy, the Solver can answer them correctly without tools; if they are too difficult, the Solver cannot solve them no matter how hard it tries, and it likewise learns nothing. The ideal training questions should fall exactly within the range of "requires tools to answer correctly."

Marvo uses Margin Reward to quantify this. Specifically, margin reward is defined as the difference in the Solver's performance when using tools versus not using tools. When the difference is positive, it indicates that tools indeed bring a gain, and the question is a valuable training sample; when the difference is zero or negative, the question is either too easy or too difficult, and its training value is limited.

This signal is fed back to the Challenger, driving it to generate questions of moderate difficulty, thereby forming an adaptive curriculum—as the Solver's ability improves, the Challenger correspondingly increases question difficulty, always maintaining a training tension of "just beyond reach."

Reinforcement Learning (RL) plays a key role here: the Solver does not learn from labeled samples, but through trial and error interacting with the environment, receiving reward signals based on whether it ultimately answers correctly, and thereby adjusting its policy. In tool-calling scenarios, the Solver needs to learn a more complex decision—not only solving the problem, but also judging "when to call a tool and which tool to call." This is known as a tool-use policy, which belongs to the class of sequential decision problems: each step's choice affects subsequent states and the final outcome. Reinforcement learning is naturally suited to such delayed-reward problems, allowing the model to gradually discover effective tool-calling patterns through extensive attempts. Marvo's margin reward is essentially a modification of the standard RL reward signal: rather than only looking at whether the final answer is right or wrong, it looks at "whether the tool brought a relative gain," thereby transforming sparse final rewards into denser, more targeted training signals.

Cross-Domain Validation: Average Accuracy Improved by 7.1%

The research team validated Marvo's effectiveness on multiple benchmarks, covering scenarios such as agent reasoning, visual search, mathematical reasoning, and general real-world reasoning. The results show that Marvo improved average accuracy by up to 7.1%.

What is noteworthy about this number is that the improvement does not come from more labeled data or a larger model, but from the design of the training signal—margin reward makes curriculum learning possible under unsupervised conditions.

Why This Is a Scalable Path

The study's most core contribution is not the score improvement on any specific benchmark, but the proposal of a mechanism for scaling problem complexity without relying on human supervision. Traditional methods either rely on manually designed curricula or on large-scale labeled data; Marvo uses the Solver's own performance gap as a signal to automatically adjust training difficulty.

This means that as long as you have unlabeled images on hand, it is possible to train a visual agent with tool-calling capabilities. For multimodal scenarios where data acquisition is costly—such as medical imaging or remote sensing images that require professional annotation—this path has particularly prominent practical value.

Of course, the stability of self-evolution methods, noise handling of reward signals, and cross-modal transferability remain questions that subsequent research needs to answer. But the idea of margin reward provides a concise and effective direction for training visual agents.

Adaptive Curriculum Learning draws on the intuition of human education: learning content should always be slightly above the learner's current level. In machine learning, curriculum usually refers to the ordering of training samples from easy to difficult. Traditional curriculum design relies on manually preset difficulty levels or heuristic rules, whereas Marvo allows the curriculum to be "adaptively" driven by the training process itself. The Challenger dynamically adjusts question difficulty based on the Solver's margin reward, equivalent to an automatic question-generation system that constantly probes the learner's zone of proximal development. This mechanism is in the same lineage as the idea of Self-Play: by having two roles of the same system compete or cooperate with each other, increasingly challenging learning signals are generated without external annotation. The difference is that Marvo's Challenger does not seek to defeat the Solver, but to generate questions that "exactly require tools," so it is less competitive and more collaborative. This design reduces the risks of mode collapse or uncontrolled difficulty that easily arise in self-play, making training more stable and controllable.

One-line Summary

Marvo uses a Challenger-Solver dual-agent setup and margin reward to train visual agents through self-evolution from unlabeled images, improving average accuracy by 7.1%.

Paragraph Summary

Visual agents can alternate between reasoning and tool calling, but training them usually relies on expensive human annotation. A new study proposes Marvo, a method for self-evolution from unlabeled images. It uses a Challenger-Solver dual-agent architecture: the Challenger automatically generates question-answer pairs, and the Solver answers them through reinforcement learning while learning to call tools. The core innovation is margin reward, defined as the difference in the Solver's performance with and without tools, used to select questions that "require tools to answer correctly" and drive adaptive curriculum learning. On benchmarks including agent reasoning, visual search, mathematics, and real-world reasoning, Marvo improves average accuracy by up to 7.1%. The method offers a scalable path to increasing problem complexity without human supervision, and is especially valuable for multimodal scenarios with high annotation costs, though self-evolution stability, reward noise, and cross-modal transfer remain open questions.