[KongchangAI]
· 1 min read· 942 words

Action Geometry Fidelity of World Models: A Study of LeWM Local Sensitivity

Action Geometry Fidelity of World Models: A Study of LeWM Local Sensitivity

Study reveals world models predict accurately but distort local action geometry, potentially causing systematic directional bias in CEM/MPC planning.

The study treats "whether world models faithfully preserve local action geometry" as an independent evaluation dimension. Using LeWorldModel in Reacher and OGBench Cube, it compares action-Jacobian Gram matrices extracted from the model with "true" geometry from simulator counterfactuals. Results: dominant action direction deviates by ~36° and ~42° under minimal perturbation; tenfold perturbation reduces geometric discrepancy by ~70% and ~47%; latent geometry varies far more across scales than physical-state geometry; and the model misestimates response magnitudes for unseen action directions. The authors stress that learning meaningful action dynamics does not equal faithfully capturing local sensitivity structure, and whether geometric deviations directly cause planning failures remains to be verified; future work will explore incorporating local geometry into CEM sampling covariance.

In model-based RL, world models are typically judged by prediction accuracy or downstream control performance. But does accurate prediction mean the model truly understands "how actions affect the future"? A study accepted as a poster at the NeurIPS 2026 NeurReps Workshop singles out this question: do world models preserve local action geometry?

reddit source: [Research] Does LeWorldModel preserve local action geometry? Implications for model-based RL and planning

Motivation: Why CEM/MPC Cares About Action Geometry

Planners such as CEM (cross-entropy method) and MPC (model predictive control) work by sampling, evaluating, and comparing candidate action sequences on a learned dynamics model. How much, and in which direction, a small change along each direction in action space alters the future state directly determines sampling efficiency and the soundness of comparisons.

If the model locally "rotates" the action response by some angle, the planner may systematically explore in the wrong direction. This is exactly the angle the paper takes: it treats the local sensitivity structure of "action-to-state" as an independent evaluation dimension, rather than looking only at overall prediction error.

The action-Jacobian Gram matrix is key to understanding the method. A dynamics model can be written as $s_{t+1}=f(s_t,a_t)$, and its Jacobian with respect to actions $J=\partial f/\partial a$ describes the instantaneous influence of each action component on each component of the next state. The Gram matrix $G=J^\top J$ then characterizes the magnitude and direction of sensitivity along each direction in action space: its eigenvalues correspond to gains along the principal sensitive directions, and its eigenvectors give those directions. Two models with similar prediction accuracy can still differ in $G$—for example, if the eigenvectors rotate, the model believes the "most effective" action direction differs from that of the true system. When comparing two $G$ matrices, the angle between principal eigenvectors is commonly used to measure directional deviation, and eigenvalue ratios to measure magnitude deviation—exactly corresponding to the "angle difference" and "response magnitude accuracy" reported in the paper.

Experimental Design: Simulator Counterfactuals as the Yardstick

The study selects LeWorldModel (LeWM) as its subject and tests it in two environments: Reacher and OGBench Cube.

The core method is to compare action-Jacobian Gram matrices: one extracted from the learned world model, the other obtained from counterfactual queries to the simulator. Simulator counterfactuals serve as the "ground truth" against which the model's characterization of local action structure is measured. The perturbation scale is also varied systematically to see whether geometric error changes with perturbation size.

The advantage of this design is that it separates "prediction error" from "structural error"—a model may perform well on overall prediction while already being off in local directions.

Key Findings: Direction Is Off, and Alignment Improves with Scale

First, at the smallest perturbation scale, the dominant action direction shows a clear deviation: about 36° on Reacher and about 42° on Cube. This means the model's "most effective action direction" does not match the true direction in the simulator.

Second, after magnifying the perturbation tenfold, the geometric discrepancy shrinks substantially: about 70% lower on Reacher and about 47% lower on Cube. In other words, the larger the perturbation, the closer the model's action geometry is to the truth; accuracy is worst under small perturbations.

Third, the action geometry measured in the learned latent representation varies far more across perturbation scales than the geometry measured in physical state coordinates. This suggests that the local structural stability of latent space is itself a concern.

Fourth, the model's local geometry is less accurate than simulator-induced geometry when predicting response magnitudes for "unseen action directions." That is, when pushed along an unseen direction, the model's estimate of the state change magnitude is biased.

An interesting conclusion the paper draws is that a world model can learn meaningful action dynamics without necessarily faithfully capturing their local sensitivity structure. The two are not equivalent.

Implications for Planning Practice: Is CEM's Covariance Enough?

The authors explicitly state that it has not yet been established whether these geometric deviations directly cause planning failures; this is the next question to answer. The direction they are interested in is whether incorporating local action geometry into CEM's sampling covariance can improve exploration and control performance.

This raises an open question: does the covariance adaptation mechanism in standard CEM already implicitly compensate for local geometric deviations? If covariance is adjusted only based on sampled returns, and the geometric deviation is a systematic directional rotation, then covariance alone may not correct it. Conversely, if sampling is dense enough, geometric error may average out. There is no consensus yet.

From an engineering perspective, this study provides an actionable diagnostic tool: comparing action-Jacobian Gram matrices can expose structural defects in a world model's local action response without relying on downstream control experiments. For teams using world models for MPC, this check may be more informative than simply looking at validation-set prediction error.

Reference: Paper "Does LeWorldModel Preserve Local Action Geometry?", NeurIPS 2026 NeurReps Workshop poster, arXiv:2610.31716.

Background Note

"Perturbation scale" here refers to the magnitude $\epsilon$ of random perturbations applied to a nominal action. Local geometry should in principle be defined under infinitesimal perturbations, but practical estimation of the Jacobian requires finite differences or regression: when perturbations are too small, numerical noise dominates; when too large, nonlinear effects make the linear approximation invalid. The paper systematically varies $\epsilon$ and finds that directional deviation is largest at small scales and converges when magnified, indicating that estimates are affected by both noise and nonlinearity. This also explains why geometry in latent space varies more dramatically with scale: latent coordinates are nonlinear, different scales correspond to different local linearization points, and the curvature of the coordinates themselves amplifies this instability.

#world models#local action geometry#action-Jacobian Gram matrix#model-based RL#CEM planning#MPC#LeWorldModel
Share:

Related articles