AGENTS.md Coding Rules Put to the Test: Code Quality Unchanged, Development Efficiency Significantly Improved

Background

Recently, a developer added coding rules—inspired by Andrej Karpathy and compiled by @jiayuan_jy—to an AGENTS.md file, then tested them across 40 PRs (Pull Requests) from the OpenClaw project on three different coding Agents. The experiment aimed to answer a simple question: does providing AI coding agents with explicit coding guidelines actually improve their performance?

Andrej Karpathy and His Coding Philosophy

Andrej Karpathy is the former AI Director at Tesla, a founding member of OpenAI, and one of the most influential educators in deep learning. His coding rules emphasize simplicity, readability, and clarity—avoiding over-abstraction, favoring straightforward implementations, keeping functions short and focused, and minimizing unnecessary dependencies. These principles are essentially a modern interpretation of "defensive programming" and the "KISS principle" (Keep It Simple, Stupid), and they're particularly well-suited as constraints for AI coding agents because they provide clear, unambiguous decision boundaries.

What Is AGENTS.md?

AGENTS.md is an emerging practice where developers place a dedicated Markdown file in the project root to communicate coding standards, architectural preferences, and workflow requirements to AI coding agents (such as Cursor, Copilot Workspace, Devin, etc.). It's similar to the traditional CONTRIBUTING.md, but designed specifically for AI Agents, helping them better understand project context and expected code style.

In simple terms, AGENTS.md is a "project manual" you write for your AI coding assistant, telling it which rules to follow, what code style to adopt, and which anti-patterns to avoid.

The rise of this practice is closely tied to the concept of "Context Engineering." From early Prompt Engineering to today's Context Engineering, the methodology of AI application development is rapidly evolving. Prompt Engineering focuses on how to phrase individual instructions for better output; Context Engineering focuses on how to systematically organize and provide information—including project structure, coding standards, historical decisions, and more—so that AI maintains consistency and efficiency throughout the entire workflow. AGENTS.md is a concrete implementation of Context Engineering in software development. It makes implicit project knowledge explicit, so AI Agents don't need to repeatedly probe and "guess" what the project expects.

Experiment Design

The core design of this experiment included:

• Test scale: 40 real PRs from the OpenClaw project
• Test subjects: Three different coding Agents
• Variable control: Comparing Agent performance before and after adding Karpathy-style coding rules
• Evaluation dimensions: Code quality, number of tool calls, completion time, and cost

About the OpenClaw Project

OpenClaw is an open-source project frequently used as a benchmark platform for AI coding capabilities. Using real project PRs as test cases—rather than artificially constructed programming problems—better reflects AI Agent performance in actual development scenarios, including contextual understanding of existing codebases, adherence to established project patterns, and compatibility with other code. This evaluation approach based on real engineering tasks is closer to developers' daily usage scenarios than pure algorithm benchmarks like HumanEval or MBPP.

This controlled experiment design ensures the results have a certain degree of credibility and reference value.

Core Findings: Quality Unchanged, Efficiency Improved

Code Quality Essentially Unchanged

The experiment results showed that after adding coding rules, the code quality generated by AI Agents remained essentially the same as before. This means the rules didn't make Agents write "better" code—at least not in terms of measurable quality metrics.

This result is actually not surprising. Current mainstream large language models (such as GPT-4, Claude, etc.) have been trained on massive amounts of code and already possess a fairly high baseline for code generation. Coding rules cannot break through the model's inherent capability ceiling, but they can help the model converge on the correct answer faster.

Significant Efficiency Improvements

The meaningful changes appeared in efficiency:

• Fewer tool calls: Agents no longer needed to repeatedly probe and correct, reducing unnecessary intermediate steps
• Lower time consumption: Time needed to complete the same tasks was noticeably shorter
• Lower cost: Overall running costs decreased due to reduced token consumption and API calls

Understanding How AI Coding Agents Work

To understand why reducing tool calls matters so much, you need to understand how modern AI coding agents operate. They typically employ a "plan-execute-verify" loop: first analyzing task requirements, then collecting information through tool calls (such as reading files, searching the codebase, running tests, executing terminal commands, etc.), then generating code, and finally verifying results. Every tool call consumes time and tokens, so reducing unnecessary tool calls directly translates to lower costs and shorter completion times. A typical complex task might involve 10-30 tool call cycles—if coding rules can reduce that by 20%-30%, the cumulative effect is quite significant.

Token Billing and Cost Impact

Current mainstream LLM APIs use a per-token billing model, where every input and output token has a corresponding price. For AI coding agents, a single complete task execution might consume tens of thousands or even hundreds of thousands of tokens—including system prompts, context file contents, multi-turn conversation history, tool call results, and more. Therefore, even if the cost per individual call seems minimal, cumulative costs become substantial in high-frequency usage scenarios. Eliminating one unnecessary tool call cycle might save thousands of tokens, which at team scale could mean hundreds or even thousands of dollars in monthly cost differences.

In other words, coding rules in AGENTS.md help AI coding agents "take fewer detours" and output results directly in the expected direction.

Why Do Coding Rules Improve AI Agent Efficiency?

Rules "Reduce Friction" Rather Than "Raise the Ceiling"

This experiment reveals an important insight: providing AI Agents with explicit coding standards doesn't help them write code beyond their capabilities—it reduces waste in their exploration and decision-making process.

When an Agent clearly knows what code style and patterns the project expects, it can reach the goal more directly instead of hesitating between multiple possible implementations. This aligns with the core principle of Prompt Engineering—giving AI more explicit constraints often produces more efficient output.

From an information theory perspective, coding rules essentially reduce the entropy of the AI Agent's decision space. Without rules, the Agent faces countless micro-decisions: "camelCase or snake_case?" "Classes or functions?" "Error handling with try-catch or Result types?" Each decision point might trigger additional context queries. Explicit rules transform these open-ended questions into deterministic instructions, thereby compressing the search space.

An Analogy

Imagine assigning a task to a newly onboarded engineer. If you only say "build this feature," they'll likely need to repeatedly confirm code style, naming conventions, and architectural choices. But if you provide them with a detailed coding standards document upfront, they can start working directly, saving considerable communication overhead. AGENTS.md has a similar effect on AI Agents.

Practical Recommendations for Developers

1. Investing time in writing AGENTS.md is worthwhile: While it won't make code better, it saves real time and money
2. Rules should be specific and actionable: Karpathy-style rules are effective because they provide clear, followable guidance rather than vague principled descriptions. For example, "keep functions under 50 lines" is more effective than "write concise code," because the former gives a definitive criterion
3. Focus on ROI rather than perfection: In AI-assisted programming scenarios, efficiency improvements may deliver more practical value than minor quality adjustments
4. Continuously iterate on rule content: As the project evolves, regularly update the rules in AGENTS.md to ensure AI Agents always receive the latest contextual information
5. Distinguish between rule tiers: Rules can be categorized into "must follow" (e.g., security-related), "strongly recommended" (e.g., architectural patterns), and "preferences" (e.g., formatting style) to help Agents make correct priority judgments when constraints conflict

Broader Implications

This experiment also hints at a direction for AI coding tool development: rather than endlessly pursuing improvements in model capabilities alone, optimizing the "interface" of human-machine collaboration may be more impactful. Good context-providing mechanisms (like AGENTS.md) can significantly improve the actual user experience and economics without changing the model itself.

This finding echoes a classic insight from software engineering: in complex systems, interface design is often more important than component performance. As Fred Brooks pointed out in The Mythical Man-Month, the greatest difficulty in software development isn't technical implementation but communication and coordination. AGENTS.md is essentially solving the "communication" problem between humans and AI Agents—it encodes a team's implicit knowledge into explicit rules that AI can consume.

From an industry trend perspective, we're witnessing a paradigm shift from "model-centric" to "system-centric." The marginal returns of simply scaling up model parameters and training data are diminishing, while unlocking existing model potential through better context management, tool integration, and workflow orchestration is becoming a more cost-effective optimization path.

For teams, maintaining a high-quality AGENTS.md file may be one of the most cost-effective AI programming efficiency optimizations available today.

Conclusion

This experiment based on 40 PRs sends a clear signal: providing AI coding agents with structured coding rule guidance is a low-cost, high-return optimization. It won't make AI smarter, but it will make AI work more efficiently—and in an era of per-token billing, fewer calls and shorter completion times translate directly into real cost savings.

If you're using AI coding agents for daily development, consider spending half an hour writing an AGENTS.md for your project. This investment will likely pay for itself after just a few uses.

Key Takeaways

• After adding Karpathy-style coding rules to AGENTS.md, AI coding agents' code quality remained essentially unchanged
• Efficiency improved significantly: fewer tool calls, shorter completion times, and lower costs
• The core value of coding rules lies in reducing Agent exploration friction, not raising the code quality ceiling
• The experiment covered 40 real PRs across three different coding Agents, providing meaningful reference value
• Providing AI Agents with structured context guidance is a low-cost, high-return optimization strategy
• AGENTS.md represents the evolution from Prompt Engineering to Context Engineering—a concrete practice in optimizing the human-machine collaboration interface