ARC-AGI-3 Experiment: Should Agents Forget or Trust Old Rules?

Minimalist agent experiment reveals: cross-level knowledge transfer mostly works, but low-confidence rules are the main cause of failure.
A developer used a minimalist agent that only counts click effects to explore the "retain or re-verify" question in cross-level knowledge transfer on ARC-AGI-3. Paired experiments show that retaining first-level knowledge performed better in 36 of 51 pairs, but in one game a key rule was verified only 5 times yet treated as reliable knowledge, causing 7 failures out of 10 attempts. Comprehensive re-verification fixes the problem, but the cost nearly cancels the benefit. The game generator further shows that the number of useless objects on the screen affects which strategy dominates. The developer advocates selective re-verification and has publicly released all predictions and code to support traceable research.
On the ARC-AGI-3 interactive puzzle game benchmark, a core question has always troubled agent designers: when an agent learns the game rules in the first level and enters the second level where the rules have changed, how much should it trust the knowledge it previously learned? Full trust lets it move fast but tend to get stuck; fully re-verifying everything is safer but may lose the advantages already accumulated. A developer shared his exploratory experiment on Reddit, using an extremely simple agent to specifically answer this question.
Experiment Design Trade-offs: Why Deliberately Avoid Neural Networks and LLMs
Unlike current mainstream approaches, this developer deliberately excluded neural networks and large language models from his experiment. His agent does only one thing: count which clicks produce actual effects in the game. This minimalist design makes failure causes traceable—every wrong decision can be mapped to exactly which "belief" went wrong.
According to the original Reddit post, the value of this approach lies in interpretability. If you use a complex LLM agent and it gets stuck on the second level, it's hard to determine whether the failure is due to model knowledge transfer, prompt issues, or exploration strategy errors. In this experiment, however, the developer can pinpoint precisely: "It only tried that rule 5 times in the first level, and in the second level it never went back to verify it again."
This trade-off also means the experiment's conclusions have limited applicability. The developer himself acknowledges that these agents are very weak compared to top LLM systems, and he doesn't claim to have discovered any major breakthrough. But as a controlled-variable experiment targeting a specific question, it has irreplaceable value.

ARC-AGI-3 is the interactive version of the Abstraction and Reasoning Corpus, proposed by François Chollet. Unlike the original ARC, which only provides static grid transformation puzzles, ARC-AGI-3 embeds puzzles in an interactive game environment: agents need to actively click, observe state changes, and infer the hidden rules of each level. Its design goal is to measure agents' generalization ability with very few samples, rather than relying on large-scale pretrained knowledge. For this reason, it is often used as a testbed for probing the boundaries of "reasoning" and "memory transfer," which also explains why in this experiment, rule verification counts and cross-level transfer strategies became core variables.
Core Finding: Retaining Knowledge Works Most of the Time, but Fragility Hides in the Details
The experimental results show a clear two-sided picture. In paired-run comparisons, retaining the knowledge learned in the first level performed better in 36 out of 51 pairs, indicating that knowledge transfer does indeed bring an advantage in most cases.
But one game broke this pattern: the agent failed 7 out of 10 attempts. The reason is specific—that key rule was only tried 5 times in the first level, too small a sample size, yet the agent treated it as reliable knowledge. When the rules changed in the second level, it neither re-verified nor abandoned this belief, directly causing repeated failures.
This case reveals an easily overlooked problem: the risk of knowledge transfer depends not only on "whether to transfer" but also on the confidence level of the original knowledge. A rule verified only 5 times and a rule verified 50 times should have different trust weights when transferred across levels. But a simple counting agent lacks this ability to distinguish.
In statistical learning and agent design, "confidence" usually refers to the estimated strength of belief in the correctness of a belief or strategy. A simple counting agent uses "number of verifications" as implicit confidence, but this proxy metric has bias: a low count may simply reflect insufficient exploration rather than the rule itself being unreliable. A more rigorous approach is to maintain a belief distribution with uncertainty (such as a Bayesian posterior) and adjust trust weights based on uncertainty during cross-task transfer. The "low-confidence knowledge" referred to in the original article is precisely an intuitive manifestation of this problem—5 verifications are statistically insufficient to distinguish accidental effectiveness from stable rules, yet the agent treated it as certain knowledge.
The Cost of Re-verification: Safe but Expensive
So what if the agent re-verifies all rules every time? The experiment shows that this does fix the above problem, but the cost almost cancels out the benefit. The cost of comprehensive re-verification is roughly equivalent to the trial-and-error cost it saves—in other words, you expend great effort ensuring safety, and in the end it's not much better than a "trust but occasionally err" strategy.
This finding points to a more practical approach: selective re-verification. Neither trusting everything nor starting over from scratch, but deciding which beliefs are worth re-checking based on signals such as the number of rule verifications and the rule's importance in the previous level. Although the developer's experiment did not directly implement this strategy, the data it provides precisely illustrates the necessity of this tiered trust.
Selective re-verification is conceptually close to the tiered strategy in the "exploration-exploitation trade-off": prioritize exploiting high-confidence, high-importance rules, and prioritize re-exploring rules that are low-confidence or sensitive to environmental changes. Common implementation signals include verification count, time interval since the last verification, frequency of rule violations, and the rule's contribution to achieving the current goal. Although the original experiment did not implement this strategy, its paired-run data happens to provide the samples needed for offline evaluation of such tiered strategies, which can be used to examine the benefit-cost curve under different trust thresholds.
Game Generator: Turning the Problem into Controllable Variables
To avoid relying solely on accidental findings in existing games, the developer also built a small game generator to deliberately reproduce the "retain or re-verify" problem. This tool revealed an interesting variable: the number of useless objects on the screen affects which strategy dominates.
When there are many irrelevant objects in the scene, the cost of re-verification rises (because there are more distractions to rule out), and at this point the strategy of trusting old knowledge is more likely to win; when there are fewer useless objects, the cost of comprehensive checking is relatively acceptable, and the advantage of the cautious strategy begins to emerge. This explains why in real games the winner between the two strategies often depends on the specific scenario rather than some universally optimal solution.
Paired runs are an experimental design method: apply two strategies (retaining knowledge vs. re-verifying) to the same initial conditions and compare their performance differences. Because the paired design controls for confounding factors such as individual game difficulty, it can better isolate the effect of the strategy itself than independent-group comparisons. The original article's statement that "retaining knowledge was better in 36 out of 51 pairs" is a typical way of reporting this design. However, it should be noted that paired comparisons can only demonstrate relative superiority and cannot be directly extrapolated to other agent architectures or more complex rule spaces, which is one reason the developer emphasizes the limited applicability of the conclusions.
One Overturned Idea and One Habit Worth Keeping
The developer specifically mentions that he had an idea that looked very promising at first but completely failed to hold up against new data. He chose to keep this failed idea in the paper because he believes it is valuable information in itself.
Another practice worth noting: he wrote down predictions before each experiment and kept all of these predictions—including a large number of wrong ones—in the code repository. This practice is not common in machine learning research, but it makes post-hoc analysis honest: readers can see which intuitions were validated and which were overturned, rather than only seeing a polished success story.
The paper and code are publicly available on GitHub (repository name: arc3-transfer-recheck), and the developer explicitly states that he hopes to receive feedback at the methodological level, especially suggestions on statistical methods. For researchers concerned with agent knowledge transfer and the exploration-exploitation trade-off, this small experiment provides a clean cross-section of the problem.
Related articles

RePlan-VLA: Progress-Aware Replanning Breakthrough for Long-Horizon Robotic Manipulation
The RePlan-VLA framework converts VLA models' short-horizon capabilities into reliable long-horizon execution through progress-aware adaptive replanning, improving success rates by 21.9% and 10.4% on RoboCasa365 and CALVIN, and validating real-world deployment on a Franka Research 3 robot.

Geometric Bridge Flow Matching: A New Breakthrough in Blind Rigid Motion Correction
Geometric bridge flow matching enables blind rigid motion correction in head CBCT, achieving 37.1 dB PSNR and 0.27 mm RPE on simulated data in ~9.1 minutes, outperforming existing methods.

Google Releases Gemini 4 Hydrogen: 2.5x Faster Without Sacrificing Intelligence
Google releases the lightweight Gemini 4 Hydrogen: 2.5x faster task completion, AA Intelligence Index of 54 beating its flagship sibling, 7.3s single-step on Agent Arena faster than Claude and GPT rivals, four thinking levels with pay-as-you-need pricing, now available across AI Studio and more.