AI Coding Assistant Fails: An Index Site Documenting Agent Incidents

When AI Coding Assistants Start Causing Trouble

As AI coding tools like GitHub Copilot, Claude, and Cursor become widespread, more and more developers are delegating code writing, refactoring, and even system operations to AI coding agents. These agents are intelligent systems built on large language models (LLMs), fundamentally different from earlier tools that merely offered code completion suggestions. Modern coding agents possess multi-step reasoning, tool use, and environment interaction capabilities — they can not only generate code snippets but also read file systems, execute terminal commands, call APIs, and even autonomously decide their next actions. This role shift from "advisor" to "executor" has expanded the impact boundary of AI coding agents from inside the editor to the entire development environment and even production systems.

However, with greater capability comes greater risk. When these agents are granted permissions to directly execute commands, modify files, and operate databases, a single misjudgment can have consequences far beyond a simple syntax error.

Recently, a project called "I Have Been Clawed" debuted on Hacker News as a Show HN post. Its mission is crystal clear: to build an index dedicated to documenting AI coding agent incidents. The project name itself is a clever pun — "Clawed" sounds like "Claude" (Anthropic's flagship AI model), while also evoking the sting of being scratched, vividly capturing the complex feelings developers experience after getting burned by these tools.

I Have Been Clawed project homepage

Why We Need an AI Agent "Incident Index"

In traditional software engineering, the postmortem is a well-established practice. This methodology originated from accident investigation frameworks in aviation and healthcare, and was later adopted by tech giants like Google and Meta for software engineering, forming a "blameless postmortem" culture. The core philosophy is that incidents are manifestations of systemic issues, not the result of individual mistakes. A standard postmortem typically includes timeline reconstruction, Root Cause Analysis (RCA), impact assessment, remediation measures, and prevention recommendations. Google even codified this practice in the classic book SRE: How Google Runs Production Systems, making it a cornerstone methodology of Site Reliability Engineering (SRE).

Large companies typically write detailed incident reports analyzing root causes, blast radius, and improvement measures. But in the emerging field of AI agents, comparable public knowledge bases are virtually nonexistent.

The Hidden and Sporadic Nature of Incidents

Errors from AI agents tend to share several characteristics:

Unpredictability: The same prompt can produce completely different behaviors in different contexts, potentially executing dangerous operations. This is a fundamental departure from traditional programs' deterministic execution — conventional code always produces the same output for the same input, while the probabilistic generation mechanism of LLMs means each execution may take a different path.
Privilege escalation: When an agent has shell execution or file system write permissions, a single wrong judgment can lead to deleted files or wiped databases. More dangerously, modern coding agents typically run in an "agentic loop" — the model repeatedly observes the environment, makes decisions, and executes actions. An early misjudgment can be amplified through subsequent loops.
Difficult traceability: The "black box" nature of AI makes root cause analysis far more challenging than with traditional program bugs. Traditional programs can pinpoint issues through logs, stack traces, and debuggers, but an AI agent's decision-making process is embedded within billions of neural network parameters. Even with Chain of Thought output, the model's true reasoning path cannot be fully reconstructed.

These characteristics mean that individual developers' hard-won lessons often get lost in the noise of social media. "I Have Been Clawed" aims to centrally archive these scattered cases, creating a publicly accessible, cautionary reference.

The Project's Core Value and Industry Significance

From Individual Cases to Failure Mode Identification

A single incident might be just a fluke, but when enough cases are aggregated, it becomes possible to identify common failure modes. For example:

Agents executing rm -rf or database deletion operations without secondary confirmation;
Agents misinterpreting vague user instructions and performing destructive operations;
Agents "taking initiative" to commit erroneous code in automated pipelines.

Once these patterns are systematically cataloged, they can provide real-world feedback for tool vendors to improve their guardrails. Guardrails are the umbrella term for technical mechanisms in AI systems that constrain model behavior and prevent harmful outputs. In the context of AI coding agents, guardrails span multiple layers: input-layer prompt filtering and injection detection, reasoning-layer behavioral policy constraints (such as prohibiting specific dangerous commands), output-layer code security scanning, and execution-layer privilege isolation and operation approval. Companies like Anthropic and OpenAI implement intrinsic constraints at the model level through techniques like RLHF (Reinforcement Learning from Human Feedback) and Constitutional AI, while IDE tools like Cursor and Windsurf add external protections at the engineering level, such as command whitelists and sandboxed execution. Feedback from real incidents is the most valuable input for iteratively improving these safety mechanisms.

At the same time, these pattern-based summaries help developers establish more reasonable usage boundaries.

A Wake-Up Call for the AI Coding Industry

Current marketing for AI coding tools overwhelmingly focuses on "efficiency gains" and "productivity revolution," while discussion of potential risks remains relatively insufficient. A public incident index effectively serves as the industry's "safety reality check" — reminding developers that AI agents still require human oversight, especially when irreversible operations are involved.

This kind of transparent incident documentation has well-established precedents in other industries. Aviation's ASRS (Aviation Safety Reporting System) and healthcare's adverse event reporting systems have both proven a key principle: systematically collecting and analyzing failure cases is one of the most effective means of improving industry-wide safety standards. The AI coding agent industry is at a critical inflection point, transitioning from "wild growth" to "mature governance," and projects like "I Have Been Clawed" are early signals of this transformation.

Safety Recommendations for Using AI Coding Agents

Drawing on the lessons reflected in this type of incident index, developers can take the following measures to reduce risk in practice:

Principle of Least Privilege

Don't readily grant AI agents direct access to production environments. Running agents in sandboxes, test environments, or restricted containers is the most fundamental line of defense.

The Principle of Least Privilege (PoLP) is one of the foundational principles of information security, first proposed by Jerome Saltzer in 1975. Its core idea is that any entity should only be granted the minimum set of permissions necessary to complete its task. In the context of AI agents, specific implementation strategies include: using Docker containers or virtual machines to provide isolated execution environments, restricting system call scope through Linux capabilities or seccomp, employing read-only filesystem mounts to prevent accidental writes, and leveraging database role-based access control to limit agents to specific tables and operation types. These measures form a defense-in-depth system, ensuring that even if an AI agent "makes a mistake," its impact is strictly contained within acceptable boundaries.

Secondary Confirmation for Critical Operations

For irreversible operations like deleting files, modifying databases, or deploying to production, always retain a human approval step — don't let agents complete these fully autonomously. In engineering practice, this is typically implemented through a Human-in-the-Loop (HITL) mechanism: the agent pauses execution when it detects an impending high-risk operation, presents the operation details, impact scope, and expected results to the human operator, and proceeds only after confirmation. Some advanced AI coding tools have already begun building in such mechanisms, for example by displaying a confirmation dialog before executing terminal commands.

Version Control and Regular Backups

Ensure all code changes are under version control with Git or similar tools, and that critical data has regular backups. Even if the agent makes a mistake, you can quickly roll back. A recommended practice is to create a dedicated Git branch before the AI agent starts working, with all agent changes committed to that branch, merging into the main branch only after human review and confirmation. This gives the agent freedom to operate while preserving a complete operation history and convenient rollback path.

AI-generated code needs to go through human review. Code that appears reasonable may harbor logic flaws or security vulnerabilities — blind trust is a breeding ground for incidents.

Multiple studies have shown that while AI-generated code excels in syntactic correctness and functional implementation, it has systematic blind spots regarding security. A 2023 Stanford University study found that code written by developers using AI assistance was actually more likely to contain security vulnerabilities, partly because the "false confidence" brought by AI assistants lowered developers' vigilance. Common security risks in AI-generated code include: insufficient SQL injection protection, hardcoded keys or credentials, insecure deserialization, and inadequate validation of user input. Therefore, subjecting AI-generated code to code review processes equally as rigorous as those for human-written code, supplemented by automated security scanning tools (such as Semgrep, CodeQL, etc.), is a critical safeguard against risk.

Conclusion: Embrace the Tools, But Respect the Risks

"I Have Been Clawed" hasn't generated massive buzz on Hacker News yet (only 6 points and 1 comment at the time of its posting), but it touches on a rapidly heating topic: In the age of AI agents, how do we strike a balance between efficiency and safety?

The project's very existence is its value — it aggregates scattered "failure" experiences into collective memory. For every developer who is handing more and more tasks over to AI, browsing an index like this may be more sobering than reading ten marketing puff pieces. Technological progress deserves to be embraced, but a healthy respect for risk should never be absent.