Matrix Plus Steps: Making Tool-Using Agents Truly Learn from Experience

An agent memory system learns only from externally validated successes, reusing experience via an associative matrix plus ordered step records.
This article introduces a novel memory system for a production-grade agent platform, whose core principle is "trust only externally validated results": each run is recorded as an episode, but only outputs confirmed by tests, tool returns, or human review count as successful, and the agent's self-assessed success never counts. Knowledge learned from episodes is stored in two-part memory—a delta-rule associative matrix for context-based procedure lookup, and addressable ordered step records for maintaining execution order and outcome counts. Simulations show external validation significantly boosts actual task solve rates; in a 100,000-episode storage test, retrieval stays at dozen-millisecond latency and never recalls invalid or others' memories, with precise recording making revisits succeed on the first try.
When agents call tools and robots execute tasks, they often fall into two traps: repeating past mistakes, and learning from runs that "seem successful" to them. To address these issues, researchers built a novel memory system for a production-grade agent platform that combines episodic memory with procedural memory, enabling agents to reliably accumulate and reuse experience.

Core Design: External Validation and Dual-Module Memory
The foundation of this memory system is "trust only externally validated results." Every run is recorded as an episode, but only outputs that pass external checks, tests, tool-returned results, or human confirmation are deemed successful; the agent's own claims never count. This principle directly addresses the pain point of "learning from false success."
Knowledge learned from episodes is stored in a two-part memory:
- Associative matrix: Written using the delta rule, it is responsible for finding the applicable procedure given the current context. It is essentially a "finder," because the matrix sums associations and does not preserve order, so it cannot return or delete an individual record on its own.
- Ordered step records: A set of addressable step records that answer "what should be done after a given step." Here, steps are the agent's goals, not specific actions. Ordering, outcome counts, and deletion operations all live in these records.
This division of labor solves the matrix's inability to maintain order while preserving its fast associative capability.
The delta rule is a classic learning rule in associative memory, dating back to the perceptron learning algorithm of the 1960s. Its basic form is: when an input pattern and an output target appear together, the connection weights are incrementally adjusted by their product (or error signal), i.e., Δw = η·x·y. In this system, the delta rule is used to write "context–procedure" associations from episodes into the associative matrix, so that similar contexts can activate the corresponding procedure. Because this rule only sums associations and does not record write order, the matrix inherently lacks serialization capability; this is precisely why an additional ordered step record must be maintained. The advantage of the delta rule is its computational simplicity and support for online incremental updates, making it suitable for high-frequency write scenarios in agent memory.
From Simulation to Real Storage: The Value of External Validation
In a simulated environment, a robot that judged its own success and stopped when it believed it was done reported completing the vast majority of tasks, yet its actual solve rate was low; after introducing external validation, its actual solve rate improved substantially. This contrast clearly quantifies the harm of "self-assessed success"—agents tend to overestimate their own performance, and external checks can correct this bias.
In a storage test containing one hundred thousand synthetic episodes (occupying hundreds of MB on disk), the system exhibited several key properties: the oldest episodes could still be found with priority, episodes from others' memories were never returned, and success records guaranteed solely by the model itself were never recalled. Query latency was only a dozen or so milliseconds on a laptop and only a few dozen milliseconds (median) on a server node. Task outcomes were measured across about ten thousand confirmed records, and on these records, precise recording made every revisit succeed on the first attempt.
These results show that scaling episodic memory does not sacrifice retrieval accuracy or speed, and that the external validation mechanism effectively filters out unreliable experience.
Episodic memory is a concept originating in cognitive psychology, proposed by Endel Tulving in 1972, referring to an individual's autobiographical memory of specific times, places, and event experiences, as opposed to semantic memory, which stores general knowledge. In artificial intelligence, episodic memory has been borrowed as an experience-storage mechanism: an agent saves each task execution along with its context and outcome as an independent record, which can later be retrieved and reused when a similar context arises. Unlike parametric learning, episodic memory does not modify model weights but instead achieves "non-parametric" experience accumulation through external storage, and thus has the properties of being interpretable, deletable, and auditable. The system in this article uses episodic memory precisely as the raw material source for procedural memory, distilling reusable procedures only from externally validated successful episodes.
Implications for Tool-Using Agents
The uniqueness of this approach lies in decomposing "procedural memory" into two complementary parts: associative lookup and ordered step records. The associative matrix quickly locates the applicable procedure, while the step records maintain execution order and outcome statistics; together they avoid the matrix's disorder defect while preserving its generalization capability.
For teams building production-grade agents, this design offers a practical reference: memory writes must be premised on external validation, otherwise the agent will continually reinforce erroneous behavior; at the same time, the memory structure must support both fast association and precise step tracking, so that agents can reliably reuse experience in complex tasks.
The system has currently been built within a production agent platform, and its file format and retrieval mechanism provide a reproducible foundation for future research. As agent application scenarios expand, this architecture based on dual episodic and procedural memory is expected to become one of the standard components of tool-using agents.
Procedural memory is likewise a cognitive science term, referring to memory of "how to do" skills and habitual operations, such as riding a bike or typing, which are usually difficult to describe explicitly in language. In the agent context, procedural memory corresponds to reusable task-execution strategies or step sequences. The innovation of this article lies in splitting procedural memory into two complementary modules: the associative matrix handles fast "context-to-procedure" matching, and the ordered step records handle "step-to-step" sequence expansion. This split draws on the idea in cognitive science of separating declarative memory from procedural memory, while avoiding the inherent shortcomings of purely connectionist storage for sequential information, providing an engineering solution for agent experience reuse that combines both generalization and precision.
One-line Summary
An agent memory system learns only from externally validated successes, reusing experience through an associative matrix plus ordered step records.
Paragraph Summary
This article introduces a novel memory system for a production-grade agent platform, whose core principle is "trust only externally validated results": each run is recorded as an episode, but only outputs confirmed by tests, tool returns, or human review are deemed successful, and the agent's self-assessed success never counts. Knowledge learned from episodes is stored in two-part memory—a delta-rule-written associative matrix responsible for finding the applicable procedure based on context, and addressable ordered step records responsible for maintaining execution order and outcome counts. Simulation experiments show that external validation can significantly improve a robot's actual task solve rate; in a storage test with one hundred thousand episodes, retrieval maintains latency on the order of a dozen milliseconds and never recalls invalid or others' memories, and precise recording makes task revisits succeed on the first attempt.
Related articles

The ARTEX Alert: How AI Agents Do Hackers' Work
FBI issues a 13-page alert on the AI agent tool ARTEX: attackers use it to automate penetration testing, with some cases achieving privileged access in under three hours. This article analyzes ARTEX's architecture, real attack cases, the authorization paradox, and AI agent security boundaries.

FBI Warns: Open-Source AI Pentesting Framework ARTEX Weaponized — From Competition Winner to Threat Dossier in Three Weeks
FBI threat bulletin warns that open-source AI pentesting framework ARTEX was weaponized — just three weeks from GitHub release to first weaponization, linked to South Korean financial breaches, supply chain attacks, and large-scale operations. Timeline, attack evidence, and defense recommendations.

Tokendoctor: The Money-Saving Solution That Cuts Claude Code Costs by 38%
Tokendoctor claims it can reduce Claude Code usage costs by 38% through token efficiency optimization. This article analyzes its principles, community reactions, and practical implications for developers.