PhysMent: Evaluating LLM Physical Reasoning with Interactive Physics Simulation

Large language models routinely score high on static science benchmarks — but when the task shifts from "read and answer" to "run an experiment," do they actually understand physics? A new benchmark called PhysMent throws LLMs into a physics simulation environment where they must actively apply forces, observe outcomes, and advance time. The results expose the real gaps in how these models reason about the physical world.

From Static Q&A to Active Experimentation

Existing science benchmarks share a common limitation: every known quantity is handed to the model upfront, and all it needs to do is perform a one-shot symbolic reasoning step to produce an answer. This design sidesteps the most fundamental aspect of physics — that information must be discovered through experimentation.

PhysMent flips this paradigm. Built on the MuJoCo physics simulator, it withholds direct access to physical quantities. Instead, models must "discover" information through a series of tool calls: applying external forces, querying object states, advancing simulation time, and modifying scene geometry. In other words, the model must act like a real experimenter — design a procedure, observe the feedback, iterate, and only then produce an answer.

This interactive, tool-mediated evaluation framework addresses a blind spot in current LLM capability assessments: whether a model's physical intuition is grounded in genuine causal understanding, or whether it has simply memorized patterns from static problem sets.

105 Scenarios and a Six-Dimensional Scoring Framework

PhysMent covers 105 scenarios drawn from classical mechanics, with a carefully structured evaluation design:

• Difficulty tiers: Two orthogonal dimensions — Easy/Hard and Single/Multi — assess both individual concepts and multi-concept combinations.
• Three scene modalities: standard scenes, object creation, and hidden objects. The hidden-objects category is especially demanding, requiring the model to actively probe the environment to uncover information it cannot directly observe.
• Scene manipulation tasks: A dedicated category tests the model's ability to intervene in and modify the physical environment.

Rather than simple right/wrong scoring, PhysMent applies a six-dimensional evaluation framework. Even when a model answers incorrectly, its exploration process, tool-use efficiency, and responsiveness to feedback are assessed separately — allowing researchers to pinpoint precisely where failures originate.

Qualitative Tasks Hold Up; Quantitative Tasks Fall Apart

The experimental results reveal a sharp dividing line. On qualitative, single-concept tasks, current models perform reasonably well, reaching accuracy rates as high as 80%. This suggests models do have some feel for basic physical concepts.

But as soon as tasks require precise numerical values and multi-step experimental procedures, performance drops dramatically. On the hardest single-concept tasks, most models fall below 30% accuracy. Crucially, the researchers found that the bottleneck here is not conceptual load — it is procedural. Specifically, models struggle with adaptive multi-step tool use: they may understand the underlying physics, but they lack the ability to systematically design and execute a sequence of experiments to converge on an answer.

Across the seven models evaluated, accuracy ranged from 25% to 67% — a substantial spread.

The Root Cause: Not Ignorance, But Inability to Experiment

PhysMent's most valuable contribution is its attribution of failure modes. The research team found that errors stem primarily from behavioral patterns rather than conceptual deficits:

• Premature answer submission: Models rush to a conclusion before conducting sufficient experimentation.
• Inefficient exploration: Tool calls lack strategy, resulting in wasted steps with little informational gain.
• Inconsistent grounding of simulation feedback: Models receive experimental results but fail to reliably incorporate them into subsequent reasoning.

This finding reframes the problem in an important way. The weakness in LLM physical reasoning is not primarily a knowledge problem — it is a behavioral one. Models lack a reliable, iterative experimental decision-making process. This aligns closely with ongoing research into agent capabilities: the real challenges lie in long-horizon planning, tool orchestration, and sustained alignment with environmental feedback — not in isolated knowledge retrieval.

Implications for AI Research

PhysMent's value extends beyond producing a new leaderboard. It functions as a diagnostic instrument. When we ask whether LLMs possess a "world model" or genuine "physical intuition," static benchmarks struggle to provide convincing answers — a model might simply be reciting physics from its training data. In a closed-loop environment that demands active experimentation, a model's full reasoning chain is exposed.

For teams building embodied AI systems, robotics controllers, or scientific discovery agents, this work points to a clear optimization target: rather than continuing to scale up physics knowledge, the priority should be improving models' experimental process capabilities — knowing when to explore, how to explore efficiently, and how to stably incorporate each step of feedback into decision-making. This may well be the critical step that takes large models from "answering questions about science" to "actually doing science."