VICO Framework: Co-Evolving Visual Environments with Vision-Language Models

The Problem with Static Training Environments

Reinforcement Learning with Verifiable Rewards (RLVR) has become the standard approach for post-training vision-language models (VLMs), but it rests on a long-overlooked assumption — that the training environment is static. The VICO paper targets this exact pain point: as the model (actor) continuously improves, fixed tasks gradually fall out of its learning frontier.

This manifests in two ways: some tasks become too easy for the model, losing their training value; others remain perpetually unsolvable. The result is a collapse in the learning signal and stagnating model improvement. It's like making a student with steadily improving skills repeat the same fixed-difficulty exercises over and over — the returns inevitably diminish.

The paper's core argument is clear: VLM post-training should allow the visual environment to evolve alongside the model itself, rather than optimizing only one side.

The core idea behind RLVR is to use automatically verifiable task outcomes — such as whether a math answer is correct or whether a target location matches — as reward signals, replacing human scoring. Compared to traditional supervised fine-tuning, RLVR doesn't require large numbers of input-output pairs; it only needs a well-designed verification function. This has yielded impressive results in areas like mathematical reasoning and code generation, and has since been adapted for VLM post-training. However, the quality of RLVR's reward signal is highly sensitive to task difficulty distribution — tasks that are too easy lead to reward saturation and vanishing gradients; tasks that are too hard mean the model almost never succeeds, resulting in sparse rewards. Both situations stall training. This tension cannot self-correct within a static corpus, making it a structural flaw in RLVR when applied to continuous training.

VICO's Co-Evolution Mechanism

VICO (Visual Environments Co-Evolving) is a co-evolutionary framework whose elegance lies in introducing a component called the Environment-as-Rewriter (EnvRewriter), which is jointly trained with the actor.

How EnvRewriter Works

The EnvRewriter's role is to edit verifiable structures on the image side — such as scene graphs, chart data tables, or protected region masks — and then re-render these structures to generate training samples with still-valid labels.

Critically, the difficulty of these new samples is calibrated to the actor's current capabilities. Calibration is based on a pass-rate-based reward: if the model's pass rate on a certain type of task is too high, the tasks are too easy; too low means too hard. Using this signal, the system continuously generates tasks of just the right difficulty, sitting at the model's current learning frontier.

A scene graph is a structured representation of image content: objects serve as nodes, while spatial or semantic relationships between objects form edges in a directed graph. For example, an image might be represented as "cat → above → table." Because scene graphs explicitly encode verifiable visual relationships, EnvRewriter can systematically adjust question-answering difficulty by modifying node attributes (such as color or quantity) or edge relationships, while using the graph structure to directly derive new correct answers without re-annotation. The same applies to chart data tables — after editing cell values, the answers to derived questions can be automatically computed. This workflow of "edit a structured intermediate representation, then render it into an image" is the technical key that allows VICO to generate valid training samples without relying on human annotation.

No Additional Human Annotation Required

The most valuable aspect of this loop is that it continuously re-aligns task difficulty with model capability — all without requiring any additional human annotation. Because EnvRewriter edits verifiable image structures, re-rendered samples come with their correct labels built in, fundamentally bypassing the dependence on large-scale human-annotated data that traditional methods require.

Experimental Results

VICO was evaluated on nine multimodal benchmarks covering two major task categories: mathematical reasoning and visually grounded understanding. Using the VICO-8B model as a reference, the paper reports the following key results:

• Up to +5.0% improvement over the base model on out-of-domain tasks
• +4.3% over the strongest self-evolution baseline
• +8.4% over the text-editing co-evolution baseline
• Comparable performance to chart-specialized RLVR methods, using 16 to 160× fewer annotated samples

That last data point is particularly noteworthy. Achieving performance on par with specialized methods while using one to two orders of magnitude fewer annotated samples directly validates the data efficiency advantage of the co-evolution approach.

Why This Approach Makes Sense

VICO's real contribution is shifting VLM post-training from "supervision via human annotation" to "co-evolution via image editing."

Traditional RLVR relies on static corpora, with a performance ceiling constrained by the scale and quality of annotated data. VICO offers a scalable new path by letting the environment dynamically generate difficulty-calibrated samples. The stronger the model gets, the more challenging the tasks the environment produces; the two pull each other forward, growing together throughout training.

This idea is philosophically aligned with recent AI training concepts like "self-play" and "curriculum learning," but VICO grounds them in the visual modality — a considerably more difficult domain to operate in. After all, editing images while ensuring label validity is far more complex than modifying text. VICO addresses this elegantly by operating on structured intermediate representations such as scene graphs, chart tables, and masks.

For researchers and engineering teams focused on multimodal large model training efficiency, VICO offers a viable direction beyond static-corpus RLVR — especially valuable in scenarios where high-quality annotated data is scarce.

Curriculum Learning refers to arranging training samples in order from easy to hard, keeping the model in its "zone of proximal development" at each stage — that is, at a difficulty level slightly beyond its current capability but achievable with effort. Self-Play goes a step further: agents generate challenging opponents by competing against themselves or past versions of themselves; systems like AlphaGo and OpenAI Five both use this mechanism. VICO's co-evolution can be seen as a fusion of these two ideas applied to the visual-language modality: EnvRewriter plays the role of a "teacher who dynamically sets problems based on the student's ability," while the pass-rate reward automatically calibrates curriculum difficulty, avoiding the subjectivity and lag inherent in manually designed curricula. Compared to the purely textual domain, the challenge in the visual modality is that image editing must simultaneously preserve rendering quality and label consistency. By manipulating structured intermediate representations rather than editing pixels directly, VICO effectively sidesteps this challenge.