Muse Wonder Lava Lamp Fail: The Benchmaxxing Questions Behind Agent High Scores

Meta launched the coding agent models Muse Wonder and Muse Wonder Full on October 12, and the official scorecard looked impressive: Wonder Full scored 57.5 on SWE-Marathon and 79.0 on DeepSWE 1.1. But the developer community BridgeMind needed only one 3D programming task—building a lava lamp with a transparent glass shell, dynamic lava, and lighting effects—to derail the new model: it produced just a red cylinder.

This seemingly "meme" test quickly escalated into a debate over whether Agent Benchmarks can actually reflect real capabilities, and even Musk dropped a pointed "Hmm" under the related post.

Why a single lava lamp can break a high-scoring model

The task difficulty behind the red cylinder

From an implementation standpoint, a lava lamp involves 3D geometry, transparent materials, lighting calculations, and dynamic deformation. Using Three.js as an example, the model needs to construct a glass lampshade, generate irregular geometry for the internal lava, and then drive shape changes through time variables or noise functions.

The key technical difficulty is this: even if such code runs correctly, the rendered image can still be completely wrong. If transparent material parameters are misconfigured, the glass becomes an opaque solid; if the lava lacks a continuous change mechanism, the internal effect degrades into a static color block. Compilers and unit tests can hardly judge these visual defects directly.

In other words, the Agent needs a visual feedback mechanism: render screenshots, identify visual deviations, and then translate visual problems into code changes. Simply increasing the number of tool calls or extending reasoning time cannot replace this step.

The officially acknowledged 3D weakness

An easily overlooked premise: the lava lamp test targets the standard Muse Wonder, while some of the standout results Meta published came from Wonder Full—there is a capability gap between the two. At the same time, Meta's official Model Card explicitly acknowledges that visual and spatial reasoning (especially 3D tasks) is a known weakness of Muse Wonder.

Meta Chief AI Officer Alexandr Wang responded that the team indeed did not explicitly train for 3D demo games, and in hindsight perhaps should have added relevant training, but he still hopes developers will actually experience Muse Wonder's Coding capabilities. This defense was immediately undercut by Musk's "Hmm."

Three.js is a JavaScript 3D graphics library based on WebGL that lets developers create interactive 3D scenes in the browser. Three.js provides core abstractions such as the scene graph, cameras, geometries, materials, and lights, and wraps low-level graphics API calls. In the lava lamp task, the model needs to combine THREE.MeshPhysicalMaterial (for glass transmission and refraction), THREE.PointLight (to simulate the internal light source), and geometry dynamic deformation based on vertex shaders or noise functions. These components have large parameter spaces, and their visual effects are hard to infer statically from code, so they pose a significant challenge for Agents lacking visual feedback.

Model Card is a standardized document provided by the model publisher alongside the model, typically including capability descriptions, intended uses, known limitations, an overview of training data, and ethical considerations. In the Muse Wonder Model Card, Meta explicitly lists visual and spatial reasoning (especially 3D tasks) as a known weakness, meaning the limitation was identified before release. The purpose of a Model Card is to provide downstream developers with selection guidance, but in practice it is easily overshadowed by leaderboard scores; the lava lamp test turns this paper statement into a tangible failure case.

Reward mechanisms determine what models are good at

Muse Wonder's performance is directly related to the technical route in which current large models increasingly rely on Post-training. In the Pre-training stage, the model learns general knowledge such as language, code, and spatial relationships from large-scale data; entering Post-training, reinforcement learning further shapes the model's behavioral policy.

Coding Agents are particularly suited to this kind of training: give the model a code repository with bugs, require it to locate the problem, modify files, and run tests—the compilation results, unit tests, and functional checks throughout the process can provide relatively clear feedback. As training scale expands, the model gradually learns to narrow the search space, select tools, handle errors, and maintain task state across multiple rounds of modification.

The Active Focus mechanism mentioned in Meta's technical article is exactly what lets the model try other approaches after hitting obstacles, recover execution, and continue pushing the task forward. But this kind of training depends on a key condition: the environment must provide useful feedback to the model. The lava lamp happens to break the feedback style familiar to traditional Coding Agents—it requires visual judgment, not compiler judgment.

Why the model degenerated into a red cylinder still requires the full execution trace to determine. It may be that it failed to build the correct spatial structure, may be improper use of graphics tools, or may be a lack of effective visual inspection after generating errors.

Agent high scores are no longer just about model capability

The invisible boost from Harness and reasoning budget

Today's Agent Benchmarks can no longer be simply understood as giving language models a test question. Take SWE-Marathon as an example: the model needs to perform multi-step operations in a code repository, and the completion rate depends not only on the underlying model but also directly on the Harness.

Harness is the execution framework built around the model, responsible for organizing tool interfaces, managing context, saving task state, and handling exceptions. Imagine an Agent running tests continuously and producing thousands of lines of terminal logs: if all of it is appended to the context, the model will gradually lose grasp of the original requirements and key errors; a well-designed Harness can automatically extract important errors, compress useless information, and maintain modification progress.

The same applies to error recovery mechanisms: if an Agent modifies multiple files and then discovers that an earlier modification has a problem, a system that can preserve checkpoints, locate the step that introduced the error, and roll back locally has a chance to avoid rerunning the entire task chain. Reasoning budget also affects scores—more tokens, more tool calls, and multiple candidate solutions mean a larger search space and more opportunities for error correction, at the cost of higher operating costs.

Meta's official Model Card states this clearly: in the SWE-Marathon evaluation, GPT-6 Astra and Claude Fable 5.1 used the Codex and Claude Code execution systems respectively, so this project is not a controlled comparison under a unified Harness; Muse Wonder Full's 57.5 score also used the xhigh reasoning tier.

These configuration differences do not automatically invalidate the scores, but they do affect how the scores should be interpreted. Comparing the reasoning and planning capabilities of base models requires unifying the tool environment, Harness, and reasoning budget as much as possible; but if what is being compared is a production-ready Coding Agent product, the performance of the complete execution system itself has engineering value. The problem is that when various system configurations are compressed into a single score on a leaderboard, it is hard for outsiders to tell whether the gain comes from model reasoning capability, execution framework optimization, or more abundant compute resources.

SWE-Marathon is an Agent evaluation benchmark for software engineering tasks, requiring models to complete multi-step bug fixes or feature development in real code repositories. Unlike single-turn code generation, SWE-Marathon tasks usually require the Agent to explore the repository structure on its own, locate problem files, modify code, and run tests to verify. The evaluation records task completion rate, but the completion rate is highly dependent on the execution framework (Harness) and reasoning budget configuration carried by the Agent. Therefore, SWE-Marathon scores are better suited for comparing the engineering performance of complete Agent systems rather than measuring the reasoning capability of a base model in isolation.

Evaluation needs more complete execution information

This requires Agent evaluations to provide more detailed execution data, including task completion rate, token consumption, number of tool calls, runtime, and failure recovery. Otherwise, similar Benchmark scores may correspond to completely different engineering experiences.

Meta itself is also studying Benchmaxxing

Interestingly, the official Muse Wonder technical article includes a dedicated "Capabilities That Generalize" section, and what it discusses is exactly the problem behind Benchmaxxing. Meta points out that even if two models achieve similar scores on common Benchmarks, their ability to handle unfamiliar tasks can still differ significantly.

To study this phenomenon, the team divided test tasks into two categories: In-distribution tasks that Post-training focuses on optimizing, and Out-of-distribution tasks that are not specifically optimized for, and compared how Muse Spark 1.3 and Muse Wonder Full perform under different reasoning budgets. According to the experimental results Meta published, Wonder Full's performance on unfamiliar tasks improved relative to the corresponding trend of the previous generation model, and Meta believes this is related to a stronger base model—improvements in foundational capability allow Post-training gains to transfer more effectively to other tasks.

But there is an evaluation design problem here: how unfamiliar are the unfamiliar tasks, really? For an Agent trained for a long time on code repair, switching to a code repository it has never seen still counts as a similar software engineering task; whereas making a 3D lava lamp introduces spatial modeling, material rendering, and visual quality judgment in addition to software development, posing a greater shift in the combination of capabilities.

Out-of-distribution generalization cannot be described by a single composite average score. The distance between different tasks, the knowledge that needs to be supplemented, and the method of evaluating results all affect model performance. Determining whether a model has severe Benchmark overfitting also requires systematic experiments—a single lava lamp failure only shows that the model did not complete the current task and cannot directly prove that it over-optimized for public question sets.

A more reasonable approach is to gradually change the task distribution: first test whether the model can fix bugs in unfamiliar repositories, then require it to build programs using APIs it has not seen, and then add visual development tasks lacking clear test scripts. This can observe capability transfer within software engineering and also check performance changes when crossing task types. Evaluation can further change environmental constraints, such as temporarily replacing tool interfaces, introducing new error messages, or requiring the Agent to design its own verification program, to test its ability to read documentation, build plans, validate assumptions, and correct errors.

Benchmaxxing refers to the phenomenon in which model developers over-optimize for public Benchmarks, raising scores without a corresponding improvement in true generalization capability. Common methods include mixing Benchmark-like questions into training data, adjusting Post-training strategies for the evaluation format, or configuring more generous reasoning budgets and a better Harness for evaluation. Benchmaxxing is not necessarily deliberate cheating, but it causes leaderboard scores to diverge from model performance on unfamiliar tasks. Identifying Benchmaxxing requires out-of-distribution testing and systematic design of task distance; a single failure case is not enough to characterize it.

Beyond high scores, the boundaries of capability still need to be seen clearly

The lava lamp failure cannot erase Muse Wonder's progress on software engineering tasks, nor is it enough to prove that Meta has engaged in serious score gaming. But developers do need to re-understand today's Agent scores: when Post-training, tool systems, and reasoning budgets all contribute to capability improvement, a high leaderboard score reflects task completion performance under a specific set of conditions. It has reference value for model selection, but the more complex the use case, the more limited the information a single score can provide.

Future Agent R&D will continue to strengthen Coding, tool calling, and long-horizon task execution. But as models gradually take on independent software engineering work, the problems they need to handle will continually exceed the scope of existing training tasks. Whether they can identify problems in unfamiliar environments, build effective verification methods on their own, and continuously correct execution strategies through feedback will directly affect how large a scope of real work Agents can take on.

A lava lamp did not provide a complete answer to Muse Wonder's capabilities, but it offered a concrete reminder: the results a model achieves on familiar tasks and the work it can independently complete in unfamiliar environments still need to be tested separately.

One-line Summary

A single 3D lava lamp test breaks through Muse Wonder's high scores, revealing the gap between Agent Benchmark scores and real capability.

Paragraph Summary

Meta's coding agent model Muse Wonder scored high on benchmarks such as SWE-Marathon, but when developer BridgeMind tested it with a 3D lava lamp programming task, the model produced only a red cylinder. This failure exposed the model's weakness in visual-spatial reasoning, a limitation Meta's official Model Card also acknowledged. The article further analyzes that Agent benchmark high scores come not only from the underlying model but are also significantly affected by the Harness execution framework, error recovery mechanisms, and reasoning budget; SWE-Marathon is not a controlled comparison under unified conditions. Meta itself is also studying the generalization gap between In-distribution and Out-of-distribution tasks, but the definition of "unfamiliar" tasks still requires more systematic evaluation design. A single score can hardly reflect true engineering capability; evaluations should provide complete execution information such as token consumption, tool calls, and failure recovery.