Should Agent Lessons Become Permanent Rules?

Writing rules for AI coding assistants can easily go from "remember this lesson" to "200 instructions piling up that no one reads." The author of IssueFlow shares a workflow where agents propose learning proposals themselves, humans approve them, and outdated rules are regularly cleaned up.

A concrete failure case

If you're running an agentic coding workflow, you've probably run into this: the AI coding agent gets stuck for an incredibly boring reason. In one of the author's recent runs, the agent failed because it lacked Jam credentials. Jam is a tool used to report bugs and provide links to triage agents, but the problem in that particular run had no Jam link to look up at all.

Any human can see at a glance that no credentials are needed here, but the agent can't—unless you tell it in advance: first judge whether this access is relevant, then worry about whether you have permission.

As a result, the agent stopped, raised a human-gate, and waited for direction. After the author responded, he naturally hoped the next agent would remember that judgment. But turning every correction into a permanent instruction creates a new problem: rules pile up, and no one checks whether they should still be there.

What a good rule looks like

"Be more careful" is not a lesson—it gives the next agent no usable test to apply.

"When an issue has no Jam link, don't block on missing Jam credentials" is much more concrete—it states the condition and the action to avoid.

The key question is: does this proposal capture something beyond a single task? A one-off workaround shouldn't quietly become a rule for all future runs.

Proposals are evidence, not instructions

The balance the author wants sits between "too specific" and "too broad." He doesn't want to stuff the agent with 200 rules in every prompt—those would drown in an ocean of instructions, and token cost is also a concern.

In IssueFlow, agents can submit learning proposals. A librarian agent collects, merges, and generalizes these proposals into readable candidate rules, along with the actual instructions to be sent to the agent.

The human operator (the author's own words: "humans, haha") reviews two things: what the rule means, and what the agent will be asked to do. They can approve, edit, or reject.

The author doesn't want to skip this step. An agent is just as likely to draw the wrong lesson from a failure as it is to reach the wrong conclusion while writing code. "This failed once" does not equal "never do this."

A useful rule should state: what error it targets, under what conditions it applies, and what behavior it requires. It should leave enough evidence to check the reasoning.

The librarian agent is a dedicated role in the IssueFlow workflow, whose responsibilities resemble those of a technical documentation maintainer or knowledge base administrator on a software team. It does not generate learning proposals itself, nor does it make final approval decisions. Instead, it handles the "intermediate processing layer": receiving raw learning proposals from agents at various stages, identifying semantically similar or duplicate content and merging it, generalizing overly specific workarounds into reusable conditional rules, and ultimately organizing them into human-readable candidate rules and accompanying agent instructions. The reason for having this separate role is that pushing raw proposals directly to humans for approval would cause information overload—many proposals are different phrasings of the same problem, or differ only slightly in wording. The librarian's involvement amounts to preprocessing and deduplication, greatly reducing the cognitive burden of human approval while ensuring that key applicability conditions are not lost during generalization.

Which stage should a rule belong to

The next step in designing a learning system is: which agents should receive these rules?

Lessons from the 'design-review' stage don't necessarily belong in the 'cleanup' stage. Rules about implementation don't need to take up space in every unrelated stage.

Proposals come from specific agents at specific stages, and the librarian processes them separately by agent when generalizing and merging. IssueFlow's review process lets the author choose a specific stage, or make the rule a general one. Stage tags make scope visible and adjustable.

This small decision has a practical purpose: keeping instructions close to where they actually do useful work. More prompt text does not equal more guidance.

Approval is not the finish line

Once approved, the process isn't over. The author wants to measure the effectiveness of each rule.

Agents report when they use a rule. The librarian can use this history to find rules that haven't been reported used for a long time, and then suggest retiring them. The author can also retire rules manually.

Rejection matters too. When the author rejects a proposal with a reason, the reason is recorded, and the librarian uses it later when handling similar proposals. Otherwise, you might spend a whole week rejecting different variants of the same idea.

Note: usage reports are a signal for investigation, not proof that the agent followed every instruction, nor proof that a certain error would definitely occur without the rule.

Rule retirement in this mechanism is not a simple "delete" operation, but a decision process based on usage data. Its core logic is: if a rule has not been reported used by any agent in actual runs for a long time, it may no longer match the current working patterns, codebase state, or toolchain version, and keeping it only increases prompt length and token cost. The librarian proactively suggests retirement based on usage history, and the human operator can also retire manually based on their own judgment. But the author specifically points out that usage reports are only "a signal for investigation"—the fact that an agent did not report using a rule does not mean the rule is useless; it could simply be a gap in the reporting mechanism, or the error it prevents may happen not to appear in the current task distribution. Therefore, retirement decisions should combine the rule's original proposal context, current workflow changes, and human subjective risk assessment, rather than relying solely on the single metric of usage frequency.

Learning also needs human judgment

The author's biggest takeaway after building this self-learning system is: it provides a way to maintain instructions. It's not a promise that agents will stop making mistakes, but it does help eliminate many recurring problems that come from real experience.

The goal is: to let useful corrections benefit subsequent work, keep scope clear, and give people a way to remove suggestions that are no longer useful. Important lessons enter the prompt, and then get retired as rules over time.

This leaves a question the author is still thinking about: from one task to the next, how many instructions should survive?

If you're using coding agents, how do you handle this now? Do you keep adding instructions, or do you review and remove them?

The author also provides an IssueFlow workflow demo, showing a real design approval and an unexpected decision in a live run.

One-line Summary

IssueFlow uses agent-nominated learning proposals, librarian organization, human approval, and retirement mechanisms to prevent coding agent rules from piling up endlessly.

Paragraph Summary

The article shares a set of agent rule management processes built by the author of IssueFlow, aimed at solving the rule bloat problem in AI coding assistants caused by "turning every correction into a permanent instruction." The process includes: agents submit learning proposals, a librarian agent collects, merges, and generalizes them into candidate rules, and a human operator approves or rejects them. Good rules need to state the error they target, the conditions where they apply, and the required behavior, and scope by agent and stage. After approval, usage is tracked, the librarian suggests retirement, and rejection reasons are also recorded for future judgment. The author emphasizes that this is not about making agents stop making mistakes, but about providing a way to maintain instruction quality, let effective corrections benefit subsequent work, and keep scope clear.