How RL Agents "Notice" a Fake World: Physical Rule Errors Boost Detection from 50% to 73%

A researcher shared a fascinating experiment on Reddit: they trained a reinforcement learning (RL) agent with a foraging objective, then placed it into a "fake world" nearly identical to its training environment — with one physical rule changed (ground grip). The finding: the agent's internal state only began meaningfully "registering" the altered environment after the physical glitch actually caused it to miss food. This offers an intriguing window into AI internal representations and environmental perception.

Experiment Design: Change One Physical Rule, Watch How the Agent Responds

The core logic is refreshingly simple. The researcher first trained an RL agent under normal physics to learn how to forage. After training, the agent was transferred to a "mirror world" — identical to the training environment except that ground grip was modified.

Crucially, the agent was never trained to "detect whether the world is fake." Instead, the researchers attached a probe to the agent's internal state and read its internal representations to determine whether the agent "knew" it was in a fake world.

Initially, the probe's accuracy was only around 50% — essentially random guessing. Even though the agent was operating in a world with altered physics, its internal state produced no distinguishable signal that could be read out.

From 50% to 73%: Consequences Create Awareness

The turning point came when the physical error produced real consequences. Once the modified grip caused the agent to slip and actually miss a piece of food, the probe's fake-world detection accuracy jumped from roughly 50% to roughly 73%.

The researcher highlighted a telling detail: no one trained the agent to notice these glitches. Only after a glitch concretely cost the agent food did the internal state begin "logging" the anomaly.

This aligns deeply with the nature of reinforcement learning. RL agents are entirely reward-driven — they care about getting food, not about the abstract question of whether the world is "real." A deviation in physical rules is effectively invisible to the agent until it affects the reward signal. Once that deviation directly causes a reward loss, the environmental anomaly gets encoded in the internal representation.

Why This Result Matters

This small experiment touches on a key question in AI interpretability research: what exactly do an agent's internal representations encode, and under what conditions?

From a probing perspective, using a linear or lightweight probe to read model internal activations and determine whether a concept is represented is a standard interpretability technique. Here, the probe target was "am I currently in a fake world?" — and the experiment showed that detection capability changed as a function of task outcomes. This suggests that a model's internal world model is not static; it is tightly coupled to the agent's goals and lived experience.

Put differently, an agent's sensitivity to "environmental anomalies" is fundamentally shaped by its reward structure. It will not proactively represent differences that are irrelevant to its objective. This provides an operational experimental paradigm for studying how AI behaves in out-of-distribution or tampered environments — and for exploring whether AI might "notice" it is operating inside a test or sandbox.

A note on probing: Probing is a common tool in interpretability research. The basic idea: extract activation vectors from an intermediate layer of a trained neural network, then train a simple classifier (typically linear) to predict whether a given external concept is encoded in those activations. If the linear probe achieves accuracy significantly above chance, the concept is considered to have some readable representation inside the model. The advantage is that the original model doesn't need to be modified — you're just "eavesdropping" on its internal state. But there's an inherent limitation: probe accuracy reflects the linear separability of the information, not whether the model intentionally represents the concept. So when the probe jumps from 50% to 73% here, the more precise interpretation is: the reward loss made the internal activations more linearly separable between "real world" and "fake world" conditions — not that the agent developed some active, subjective awareness of its environment.

Open Source and Reproducible: Anyone Can Run It

To the researcher's credit, this experiment was released in an open, reproducible form. According to the original post, the code is open source at ItaSoRL (https://ilevytate.github.io/ItaSoRL/). The author also created a Colab notebook so that people without powerful hardware can run the code directly in the cloud, lowering the barrier to reproduction.

The author also noted that the paper is still in progress, with plans to upload the agent's internal "brain state" data for further community analysis.

Caveats Worth Keeping in Mind

This is still a preliminary experiment from an individual researcher with the paper not yet published — a few points warrant caution.

First, the sample and setup are fairly simplified: only a single physical rule (grip) was changed, and the detection accuracy figures (50%, 73%) need validation under more rigorous statistical conditions, more random seeds, and proper controls before their robustness can be confirmed.

Second, there is interpretive room around whether a "higher detection rate" equals the agent "knowing" the world is fake. More accurately, the internal representations became more separable after reward loss — not that the agent developed some form of subjective awareness.

Third, until the full paper is published with complete methodology and data, this result is better treated as a thought-provoking experimental phenomenon rather than a settled finding. That said, it presents a clear causal chain at very low reproduction cost — showing how reward loss feeds back to shape internal representations — which is valuable in its own right.

A note on world models: The concept of an internal world model has a specific meaning in RL research: it refers to the representation of environment dynamics that an agent learns (implicitly or explicitly) through interaction, used to predict action outcomes and assist planning. In model-free RL, the world model typically exists implicitly in the neural network weights rather than as a separate, queryable module. This experiment reveals an important property: this implicit world model is not a complete, objective representation of the environment — it is reward-centric, trimmed to fit the objective. Only environmental information relevant to reward acquisition gets fully encoded. This connects directly to a core concern in out-of-distribution (OOD) generalization research: when the environment changes, an agent's internal model may remain silent about deviations that "haven't caused losses yet," only updating its representations after actually paying a cost. In safety-critical scenarios, this could constitute a significant risk.

Summary

An RL agent only "registers" environmental anomalies in its internal state once the altered physics actually causes a reward loss — not before.