Agent Memory Architecture in Practice: From Session Context to Team Knowledge Governance

A Counterintuitive Question: Should Agents Have Memory at All?

Many teams building Agents default to a common assumption: the Agent should "understand me better over time," so it should continuously accumulate memory. But as projects scale up, a familiar set of problems starts to surface:

• Previously recorded knowledge has become outdated
• Different Agents hold inconsistent memories about the same thing
• Agents make wrong decisions based on stale memory
• The team has no idea why the Agent gave a particular response

This leads some to propose an alternative: the real owner of memory shouldn't be the Agent itself, but the external Harness (execution framework). Each Agent run is a relatively "clean" executor, with the Harness dynamically injecting the required knowledge, rules, and context based on the current task. Instead of storing business knowledge long-term, the Agent loads it on demand.

That said, framing the problem as "should Agents have memory" is itself a distortion. An Agent is essentially an LLM + a Harness, so treating them as peers in a discussion isn't rigorous. The notion of "statelessness" is also too absolute. The real question is how to design a layered memory governance system.

Memory Layers: From Session Context to Team Knowledge

If you were designing a team-level AI development platform, where should long-term memory live? In the Agent itself? In a Git knowledge base? In a Skill? The answer is: none of the above exclusively — instead, use layered management.

Three Types of Memory with Different Lifecycles

Persistent Knowledge (Git knowledge base): Records long-term, explicit, governable team knowledge — business domain knowledge, architecture documentation, engineering standards, project decisions, troubleshooting experiences, and reusable workflows. Its defining traits are versionability, reviewability, and traceability. This layer also captures current execution progress: which node has been reached, which tasks are complete, which tests failed, and which version of a solution is currently in use.

Session Memory: Short-term context within the current conversation — requirements the user just added, attempts already made in this session, and tentative hypotheses formed along the way. Its lifecycle typically follows the session or task and dissipates when the task ends.

Memory (cross-session memory): Scoped by user or business dimension — a user's long-term areas of interest, preferred phrasing, or favored output formats.

This layered design maps to distinct storage and retrieval mechanisms in practice. Persistent knowledge is typically stored as documents or structured data in a Git repository or vector database, recalled on-demand via RAG (Retrieval-Augmented Generation) based on semantic relevance, and then injected into the Prompt. Session memory is held directly in runtime memory or short-lived caches and destroyed with the session. The cross-session Memory layer sits between the two — usually stored in a structured database and retrieved by user ID or business entity. Each layer has its own write and expiration strategy. Mixing them in the same storage medium or omitting expiration policies leads directly to the "stale memory poisoning decisions" problem described at the outset.

Everything Eventually Gets Assembled into a Prompt

Whether short-term or long-term, all memory eventually gets assembled into a Prompt and delivered to the Agent. A complete user Prompt typically contains:

• Current rules from the Harness
• Knowledge retrieved from Git
• Current task state
• Conversation history from the session
• Relevant content from Memory
• The user's current input

The value of this layered design is clear: information with different lifecycles is cleanly separated, avoiding the "everything in one pot" style of memory contamination, and making the source and validity period of every piece of knowledge fully traceable.

When Memory Conflicts: Who Has the Final Say?

Layering introduces a thorny new problem. Consider the following conflict:

• Git architecture doc: Order API still uses V1
• Task state: The current task has already been developed using the V2 plan
• Session memory: The user just said "don't upgrade to V2 for now"
• Long-term Memory: Previous records indicate the user tends to prefer adopting newer versions first

Which source should the Agent trust?

The answer isn't "always prioritize one fixed layer" — it's that conflicts should trigger a human intervention mechanism. The full handling flow is: once multiple sources enter the context, detect the conflict first, then assess the type and scope of impact:

• If the system has a clear rule in place and the risk is low, the Agent can resolve it autonomously
• If no clear rule exists, or the Agent judges it as high-risk, pause and request human judgment
• After human intervention, log the issue and update the rules so the Agent can handle it independently next time

Regardless of how the situation is resolved, conflicts must always be instrumented and reported to enable downstream alerting, feedback, and post-mortems. This is fundamentally a positive feedback loop: execute → discover → human intervention → rule evolution.

Conflict Detection: How Structured Rules and Semantic Understanding Work Together

Conflict detection itself can be error-prone. For example, a document says "use V1 by default, use V2 for special merchants," and the current task is development for a special merchant. On the surface these two statements don't conflict — but combining them with the business condition yields the correct conclusion: "use V2."

So should conflict detection rely primarily on structured fields and version rules, or on LLM semantic understanding?

Version Rules Provide Determinism

Version rules should take precedence. All document updates should carry version identifiers, and even without explicit version numbers, Git provides a traceable commit sequence.

But be careful: sequence doesn't automatically mean "latest" or "should be used." What actually governs whether a version should be used is a combination of approval status, activation status, the scope of the knowledge, and its dependency relationships. Structured rules are primarily responsible for assessing determinism — whether two entries refer to the same knowledge object, whether one has been approved, and whether it is formally active (some knowledge may be in a partial rollout state).

In practice, version rules are typically implemented using Semantic Versioning (e.g., v1.2.3) or Git commit SHAs to identify the change history of a knowledge object. The three-part structure of Semantic Versioning corresponds to breaking changes, new features, and backward-compatible bug fixes respectively, allowing the Harness to quickly assess compatibility between two versions without reading the full content. For knowledge documents without explicit version numbers, Git commit timestamps and branch attribution can serve as substitutes — provided the knowledge base's commit discipline is strict enough. Arbitrary force pushes or squashed history will break the traceability chain.

LLMs Supplement with Semantic Judgment

The LLM's role is to surface conflicts at the semantic layer. For example, two document versions may differ in number but be entirely non-contradictory — just worded differently but expressing the same meaning. Or a document may carry applicability conditions that require the model to read and interpret before determining relevance.

The standard processing flow is: the Harness first uses rules to determine whether a knowledge item should be used directly, with automatic instrumentation at this step; if a conflict is detected, the LLM performs semantic conflict detection; this step will often require a human decision.

When a Plan Has a Flaw: Stopping Doesn't Mean Doing Nothing

Consider an even more extreme scenario: while executing a coding task, the AI discovers an obvious flaw in the already-approved technical plan — but the repository rules require execution to follow the approved version. Should the AI continue coding against the flawed plan, or immediately pause and raise a change request?

The key insight: the Agent cannot judge this on its own — it must rely on external rules. The prerequisite logic is to evaluate against the risk and interruption rules pre-configured in the Harness:

• Exception falls within authorized scope and can be safely handled → automatically continue
• More information is needed → pause and proactively extract knowledge for clarification
• Outside authorized scope or judged as high-risk → immediately stop changes and request human intervention

What "Stop" Actually Means

An important clarification: stopping changes does not mean doing nothing — what stops is irreversible execution, like deleting data. In fact, even if the Agent believes a plan is flawed, it can still continue coding — opening a new branch, or coding without committing, and waiting for human review. Once approved, the code can be pushed directly. If it turns out the plan really was wrong, the code is simply discarded. If it was a false alarm on the Agent's part, the code output is preserved while the rules get updated.

Not Every Issue Needs a Rule Update

Not every issue that arises warrants a rule update. Some are one-off events — an incomplete requirements description this time around, or a dev environment anomaly that triggered a false judgment. Only experiences that recur, have significant impact, and are generalizable deserve to enter the Harness change process.

Codifying Team Rules: Govern Them Like Code PRs

The final core question: how do you determine whether a problem is worth codifying into a team Harness rule, rather than treating it as a one-time, project-specific lesson?

The answer: Harness rules are the team's engineering code. They cannot be merged into the main branch just because one Agent or one person thinks they're useful — there must be a promotion mechanism.

The full governance workflow resembles code development:

1. An issue occurs during a task; the Agent submits a rule change request automatically, or a human submits one manually
2. The team reviews whether the change is worth promoting to a team-level rule
3. Evaluation is conducted with the new change in place
4. After going live, the rule is progressively rolled out via gradual release
5. Once validated at full scale, it's officially merged into the main branch

The team review scoring process can mirror an evaluation rubric — for example, five, four, or three points — where each score's meaning is defined by the business team.

This governance philosophy closely resembles "progressive delivery" and "Feature Flag" mechanisms in software engineering. Gradual rollout means the new rule only applies to a subset of tasks or users; its impact on Agent decision quality is observed through controlled comparison before a full rollout is decided. The evaluation stage is analogous to automated testing in CI/CD: a Golden Set of standard question-answer pairs covering typical scenarios must be prepared in advance, and both old and new rules are benchmarked and compared. This mechanism effectively guards against "rule inflation" — the phenomenon where rules accumulate endlessly, interfere with each other, and make Agent behavior increasingly unpredictable — while also providing clear version anchors for rollbacks.

Summary: The Positive Feedback Loop of Agent Governance

Putting it all together, a mature team-level Agent memory and rules system should have several core characteristics:

• Layered management: Knowledge, state, and memory are separated by lifecycle, then unified into a Prompt assembled and injected into the Agent
• Rule-driven resolution: Existing rules determine whether to resolve automatically or pause for human intervention, with full instrumentation throughout to ensure observability
• Human oversight of rule evolution: Team Harness rules cannot be auto-upgraded by AI — they must go through a review, evaluation, gradual rollout, and full-release process analogous to code PRs

The essence of this system is transforming the Agent from a "black box that increasingly knows me" into a governable, traceable, and evolvable engineering system. And an open question worth pondering: in the future, will AI be submitting only code PRs — or will it eventually be able to submit PRs for the team's own development standards as well? And who should hold the final merge authority? That may be the defining question every team building an AI development platform will need to answer.