CO₂Jump: A Self-Correcting Sampling Method for Consistent Joint Text-Image Generation

In joint text-and-image generation tasks, there's a long-overlooked problem: a model can accurately describe the correct solution to a maze in words, yet draw a completely different path in the corresponding image. Each modality makes perfect sense on its own — but together, they contradict each other. A NeurIPS paper from a collaboration between Google, Google DeepMind, and Stony Brook University directly addresses this "cross-modal inconsistency" problem with a new sampling method called CO₂Jump.

Why Joint Generation Leads to "Say One Thing, Draw Another"

Intuitively, if you ask a model to simultaneously output a text answer and a corresponding image, the two should remain consistent. In practice, that's not what happens. The researchers point out that generating text and images in parallel doesn't automatically guarantee semantic alignment — the text branch and the image branch evolve independently during sampling, with no effective mutual constraint. The result is "text correct, image wrong" (or vice versa).

This inconsistency is especially damaging in tasks that demand strict logical correspondence. Take maze solving: the right metric isn't just whether the text answer is correct, but whether the text answer and the generated image are both correct simultaneously. The paper calls this metric joint accuracy — a generation only passes if both the text and image outputs are correct. This raises the bar significantly for cross-modal coordination.

The Core Mechanism of CO₂Jump

CO₂Jump is designed so that the image generation process "listens to" the text branch. Specifically, it uses text confidence and cross-modal attention to guide image updates during sampling, gradually steering the image toward the correct result described by the text.

Even more important is its self-correcting capability. In traditional diffusion or masked generation, once a token is committed, it's difficult to revise. CO₂Jump allows low-confidence tokens to be re-masked and regenerated, meaning early decisions can be corrected as generation progresses. This "undoable" property is exactly what gives the method its name: a self-correcting coupled Markov jump process.

From an engineering standpoint, CO₂Jump requires only one model forward pass per denoising step, keeping computational overhead manageable. More importantly, the sampler itself requires no additional training — in experiments, all compared sampling methods share the same task-fine-tuned model. In other words, CO₂Jump is a plug-and-play inference-time improvement that enhances consistency without retraining.

---

The name "coupled Markov jump process" comes from stochastic process theory. A Markov Jump Process (MJP) is a class of stochastic processes where the state undergoes discrete jumps in continuous time — unlike the continuous noise typical of diffusion models, MJPs more naturally correspond to the masking and unmasking of discrete tokens. In masked generative models (such as MaskGIT-style architectures), the generation process can be modeled as starting from a fully masked state and progressively "revealing" tokens until a complete sequence is produced. CO₂Jump's innovation is to couple the text and image generation chains within the same jump process rather than letting them evolve independently — the confidence signal from the text branch influences the jump direction of the image branch through cross-modal attention, creating mutual probabilistic constraints between the two chains. Allowing tokens to be re-masked (i.e., jumping from "revealed" back to "masked") is a manifestation of the bidirectional nature of jump processes, something that standard unidirectional denoising pipelines do not permit.

---

Evaluation Tasks and Datasets

To systematically validate the method, the research team evaluated across three types of tasks: image editing, maze solving, and nonograms. What these tasks share is that the correctness of text and image can be measured jointly and objectively.

Alongside these tasks, the team also released three datasets:

• JEdit-1M: For image editing tasks, at million-scale;
• JMaze-200K: A maze-solving dataset with approximately 200K samples;
• JNono-200K: A nonogram puzzle dataset with approximately 200K samples.

These datasets provide standardized benchmarks for joint evaluation of text-image consistency and correctness, and leave room for future researchers to reproduce and extend the work.

---

A Nonogram (also known as Picross) is a logic puzzle: in a grid, each row and column is labeled with numeric clues indicating the lengths and order of consecutive filled cells. Players must infer whether each cell should be filled based on these constraints, ultimately revealing a pixel image. This type of task is highly amenable to joint evaluation: the text output can be solving steps or a sequence of fill coordinates, while the image output is the corresponding visual grid — both have a single objective correct answer that can be verified automatically. Compared to maze solving, nonograms involve more complex constraint propagation, requiring the model to maintain logical consistency between rows and columns across the entire grid. This makes them a more demanding test of cross-modal coordination, and makes "text correct but image wrong" inconsistencies much easier to expose.

---

Experimental Results: The Only Sampler with Monotonic Improvement

The most compelling finding from the experiments appears on the puzzle benchmarks. Across a range of 8 to 512 sampling steps, CO₂Jump is the only sampler among all baselines to achieve monotonic improvement on both editing quality and semantic grounding simultaneously.

This matters a great deal. Many sampling methods exhibit quality fluctuations or even degradation as the number of steps increases, whereas CO₂Jump demonstrates a stable, predictable performance curve — more steps mean better results, without sacrificing the other dimension. For real-world deployment, this kind of controllability is often more valuable than a single-point peak score.

Limitations and Future Directions

The authors maintain an open stance, actively soliciting suggestions for other tasks that can jointly evaluate text-image consistency and correctness, and expressing willingness to discuss the method, evaluation design, and limitations. This suggests that while the three current task categories are cleverly designed, their coverage remains limited — they are all structured tasks with clear correct answers. For more open-ended, subjective generation scenarios (such as creative illustration or free-form question-answering with images), defining and measuring consistency remains an open problem.

Overall, CO₂Jump offers a training-free, computationally lightweight approach that introduces cross-modal self-correction into the joint generation sampling process. For researchers and engineers focused on the reliability of multimodal models, this is a direction well worth following.