NVIDIA Paper Wins ICML Spotlight Despite Alleged Code Bugs — And the Community Is Not Happy

A robotics world model paper called DreamDojo, published by NVIDIA and built on top of Cosmos 2.5, has sparked fierce debate in the Reddit machine learning community. Despite earning ICML Spotlight recognition, the work has been called out by researchers who claim its pre-training, post-training, and evaluation code all contain serious bugs — and that the paper's own results had already been showing warning signs. The incident has once again put the reliability of top-conference peer review squarely in the crosshairs.

How It Unfolded: From Reproduction to Bug Discovery

According to the Reddit post, DreamDojo is NVIDIA's robotics world model built on top of Cosmos 2.5 — a prior model with roughly 100 citations. The research team claims to have collected approximately 44,000 hours of human demonstration data for pre-training, using 256 H100 GPUs for training. The authors are well-known and respected figures in the field with extensive peer-reviewed publication records.

The trouble started during reproduction. A researcher attempted to run post-training on the GR1 dataset released by the team and was initially able to reproduce the paper's results. But then a colleague — aided by Claude — discovered a critical bug in the post-training code that fundamentally broke the logic of the entire post-training pipeline. Following that thread, they dug into the project's GitHub Issues and found two additional bugs reported by the community, bugs that directly affect the pre-training phase.

In other words, according to this researcher, all three stages — pre-training, post-training, and evaluation — may be broken in NVIDIA's publicly released codebase. They emphasized that the code was written very poorly: it doesn't look AI-generated, but rather like something hastily written by hand.

Cosmos 2.5 is an iteration of NVIDIA's World Foundation Model series, designed to provide high-quality video generation and environment simulation for embodied AI applications like robotics and autonomous driving. The core idea behind world models is to teach AI systems the underlying rules of the physical world so they can predict future states from actions — providing a cheap "virtual simulation environment" for reinforcement learning and robot policy training. DreamDojo fine-tunes on top of Cosmos 2.5 for domain-specific robotics manipulation, aiming to improve world modeling fidelity in that setting. The value proposition: pre-train on large-scale human demonstration data, then post-train on task-relevant data to better model robot-object interaction physics and support downstream policy learning.

The Warning Signs Were Already There

Setting aside the code issues entirely, the paper's results were already raising eyebrows. The poster noted that Table 4 in the paper shows an extremely marginal performance gain — roughly 0.5 dB improvement in PSNR over Cosmos 2.5.

That number looks particularly striking given the scale of investment: 44,000 hours of human data, hundreds of hours of other data types and robot data, and 256 H100 GPUs — all to produce a barely measurable improvement. As the poster bluntly put it: "Doesn't that seem suspicious?"

They noted they didn't mind the lack of novelty, nor the fact that the data wasn't open-sourced. What genuinely shocked them was this: once they discovered the fundamental code errors, those "suspiciously tiny" results suddenly made sense. In other words, the bugs likely explain why the model showed almost no meaningful improvement. And yet neither the authors nor the reviewers apparently flagged the wildly disproportionate cost-to-benefit ratio.

PSNR (Peak Signal-to-Noise Ratio) is one of the most widely used objective metrics for image and video quality assessment, measured in decibels (dB) — higher values mean predicted frames are closer to ground truth. In video prediction and world modeling, a 0.5 dB improvement is an extremely marginal gain; meaningful algorithmic advances typically yield jumps of several dB. It's worth noting that PSNR doesn't fully capture perceptual quality, but it remains an important reference metric for evaluating world model prediction fidelity. Pairing such a tiny metric gain against a 44,000-hour dataset and 256 H100 GPUs — where even a single GPU rents for roughly $2–3/hour, meaning hundreds of thousands of dollars in compute — represents a cost-benefit imbalance that should have triggered basic plausibility checks.

Two Hard Questions: Did the Authors and Reviewers Drop the Ball?

The poster's critique cuts to the heart of academic publishing norms:

First, how did the authors not notice? Pouring enormous resources and data into a project that yields almost no measurable performance improvement is itself a major red flag. An experienced team should have questioned this imbalance during any reasonable post-mortem and gone back to scrutinize their own code and pipelines.

Second, how did the reviewers not catch it? This paper received ICML Spotlight — a fairly high honor — meaning it passed through multiple rounds of careful review. Yet neither the anomalous experimental results nor the community-reported bugs in the public codebase appear to have raised any alarm during the review process.

What this incident reveals is a potential blind spot in top ML conference review when evaluating "foundation model" papers backed by large companies, prestigious teams, and massive compute budgets. When the halo effect is strong enough, do reviewers still push back on details that defy common sense?

ICML Spotlight is a recognition tier at the International Conference on Machine Learning that sits above a standard acceptance but below an Outstanding Paper award. It is typically granted to roughly the top 5% of accepted papers, signaling that the program committee views the work as particularly original and impactful. Spotlight papers often receive oral presentation slots and enjoy greater visibility and citation potential in the research community. This is precisely why a Spotlight paper's credibility coming under scrutiny has implications far beyond the paper itself — downstream work built on these findings faces systemic reliability risks, and the conference's endorsement mechanism as a whole suffers a credibility hit. That's why this incident generated far more community response than a typical code bug report would.

The Deeper Industry Concerns This Surfaces

To be clear: the above is currently based primarily on a single whistleblower's account on Reddit. The conclusions remain to be verified by NVIDIA and independent reproducers. But even so, the discussion surfaces several questions the entire AI research community should be sitting with.

First, quality anxiety in the foundation model arms race. When "yet another foundation model" becomes the norm, is the real research value being buried under narratives of massive data and compute? A 0.5 dB improvement packaged into a Spotlight paper reflects an evaluation system that over-rewards scale.

Second, the importance of code reproducibility. It was precisely because NVIDIA released the source code that the community had any chance of finding these problems. This underscores the critical role of open-sourcing and reproducibility in maintaining research integrity — closed-source work doesn't even get the chance to be questioned.

Third, a stress test for top-conference review mechanisms. As submission volumes surge and model scales balloon, it becomes increasingly difficult for reviewers to deeply validate every paper — especially work involving large-scale training that can't realistically be independently reproduced. This structural tension is quietly eroding the credibility of top-conference honors.

Regardless of how the DreamDojo story ultimately resolves, it serves as a warning for the field: the triple endorsement of a prestigious institution, well-known authors, and a top-tier venue is not a guarantee of correctness. Real scientific scrutiny always has to come back to the data, the code, and the logic itself.