A New Threat in the AI Agent Era: How Just Reading a README Can Compromise You

GitHub is a treasure trove for developers — packed with ready-made tools for nearly every need. A time-tested rule of thumb: search before you build. Finding an existing solution saves enormous time and cost. But there's no such thing as a free lunch. Open-source software is a gift, yet some seemingly harmless tools carry a price you can't afford to pay.

Downloading and running code from strangers has always carried familiar risks: dependency poisoning, backdoors, data theft, and cryptomining malware. But with the rise of AI Agents, the threat has evolved — attacks are easier to pull off, harder to defend against, and have now reached a terrifying new milestone: getting compromised just by looking.

From "Run to Get Infected" to "Look to Get Infected"

In July of this year, a security firm disclosed a tracking operation that uncovered approximately 7,600 malicious GitHub repositories — over 800 of which were directly disguised as AI plugins. These fake plugins and tool servers had been downloaded more than 14 million times in total. Researchers didn't even need to point agents directly at these projects; they simply instructed an Agent to find a tool with a certain capability. Claude Code, Gemini, and ChatGPT all independently surfaced the malicious repositories.

What makes these repositories truly terrifying is this: you don't need to clone and run them locally. The moment an AI reads the project — even just glances at the README — you may already be compromised.

This sounds like a made-up horror story. For a long time, the conventional wisdom was "reading is safe, running is risky." That's no longer true. AI Agents now have execution capabilities. Once malicious text enters their context window, they may be unable to distinguish your instructions from instructions embedded in a webpage — because to the model, it's all just text.

Indirect Prompt Injection: From Academic Concept to Real-World Threat

This attack technique has a name: Indirect Prompt Injection. A few years ago it was largely confined to academic papers. Today it has become one of the primary security threats facing AI applications.

This isn't hypothetical. According to a Bilibili content creator who shared his experience, he personally fell victim on the evening of August 30th of this year: he asked an Agent to search GitHub for a solution, and it surfaced two repositories with 204 and 194 stars respectively — looking perfectly legitimate. The Agent, following normal procedure, read the search results into its context window. The result: an immediate error, and the entire task was killed.

Similar cases have appeared frequently across major communities:

• Brave's security team demonstrated hiding text invisible to the human eye inside a screenshot. When a browser Agent read the image, it complied — actually opening Gmail and sending an email subject line to an attacker-controlled URL. According to their report, as long as the browser has a saved login session, attackers could execute actions like repository deletion, lateral movement, supply chain compromise, CI/CD poisoning, and cryptocurrency wallet theft.
• Microsoft experienced a zero-click incident: users clicked nothing. When Copilot retrieved internal documents, it encountered hidden instructions and leaked private content it had access to.
• Anthropic used Agents to red-team 123 test cases. In autonomous mode, the attack success rate exceeded 20%. With mitigations applied, that dropped to around 10%. With the latest protection mechanisms, it fell below 0.1% — demonstrating the problem is not unsolvable.

The key distinction between indirect and direct prompt injection lies in the attack vector: direct injection involves a user entering malicious instructions to bypass model restrictions, while indirect injection involves an attacker embedding instructions inside third-party content that an AI will proactively read — webpages, documents, code comments, READMEs, OCR results from images, or even invisible Unicode characters. When an AI Agent executes a task, it automatically retrieves this external content and loads it into its context window. The model has no structural way to distinguish "instructions from the user" from "instructions embedded in content" — because both are simply part of the same token sequence. This flaw stems from the fundamental mechanism of large language models: they are trained to "follow instructions present in the context" without any built-in system for marking trusted sources. Prior to 2023, this attack was largely confined to academic literature. With the widespread adoption of Agent tools capable of file read/write and network access — such as Claude Code, Cursor, and Copilot — the attack surface expanded from "read" to "read + execute," dramatically raising the severity level.

Why You Can't Expect AI to Defend Itself

Many people ask: why not let the all-powerful AI detect and resist these attacks on its own?

The answer: it's been tested, and the results are poor. A paper co-authored by 14 research labs tested 12 injection defenses and found a breach rate of around 90%. Human red-teaming exercises broke through all of them.

This is precisely why OpenAI's Lockdown Mode help page is refreshingly blunt: it doesn't attempt to block injected content — it focuses solely on cutting off the pathway for data exfiltration. The defense priority should be structural constraints, not trusting the model to self-identify threats.

Security researcher Simon Willison distilled the root of the risk into a "fatal triad": private data, untrusted content, and outbound communication. When all three are present, an incident is only a matter of time. The solution is to eliminate one element — and the easiest one to remove is "outbound data transmission." Meta holds a similar view: you can have at most two of the three. If all three are present, don't let the Agent run autonomously.

OpenAI's Lockdown Mode is a security setting introduced for Agent scenarios in products like ChatGPT. Its design philosophy reflects the current mainstream consensus in the industry: rather than having the model semantically judge whether a piece of text is a malicious instruction (which has proven highly unreliable), it's more effective to architecturally block the channel for data exfiltration at the system level. Concretely, Lockdown Mode restricts the Agent from calling external URLs and prevents user data from being written to third-party services. Even if the model receives an injected instruction like "send file contents to this URL," the execution layer will intercept the action. This type of "execution-layer protection" is more reliable than "semantic-layer detection" because it doesn't depend on the model's comprehension — it relies on deterministic rule-based checks. The trade-off is reduced functionality: many legitimate tasks requiring internet access or external API calls are also affected. As a result, this mode typically requires users to manually enable it for high-sensitivity tasks.

The Current State of Protection in Mainstream Tools

Built on the principle of "structural constraints first," the defense logic of mainstream Agent tools has largely converged:

• Claude Code starts with read-only permissions by default; web access requires user approval;
• GitHub Copilot deploys a firewall at the cloud level;
• Cursor sandboxes commands that don't pass a whitelist before execution.

Approval-based protections are typically enabled by default, while sandbox mechanisms often require manual activation. Even DeepSeek^Ad has added relevant protections, including blocking internal network addresses and rejecting cross-origin redirects.

That said, security has always been an arms race. How much protection these measures actually provide is genuinely hard to say.

In this context, a "sandbox" refers to a technical mechanism that isolates the environment in which an Agent executes commands from the host system. Traditional sandboxes achieve isolation through virtual machines or containers (such as Docker), ensuring that even if malicious code is executed, it cannot access the external filesystem or network. The sandbox implementations in tools like Cursor are more lightweight. The core logic is: for shell commands that don't pass whitelist review, they are not executed directly in the user's local environment — instead, they run in a restricted subprocess or cloud environment, with results returned for user review before any changes are committed. Whitelist mechanisms require users or administrators to pre-declare which domains, commands, and file paths are trusted; the Agent can only act autonomously within those boundaries, and any out-of-scope action must pause for human confirmation. The shared downside of both mechanisms is high configuration overhead and a poor out-of-the-box experience — which leads many users to disable them in practice, rendering the protection meaningless.

Three Practical Tips for Everyday Users

We can't simply stop using GitHub out of fear. Drawing on established industry practices, here are three things to keep in mind:

1. Prefer traceable sources: Stick to libraries from well-known organizations or with verifiable open-source provenance. Avoid projects with unclear origins.
2. Separate high-risk permissions: Don't let a single Agent simultaneously hold "access to private data + handling of untrusted content + outbound request capability." For sensitive tasks, cut outbound requests first.
3. Gate your Agent: Implement field filtering, length limits, and whitelist mechanisms to constrain both what the Agent can access and what it can output.

In this era of rapid change, we've gained unprecedented capabilities — and encountered dangers that were previously unimaginable. As AI Agents evolve from "reading text" to "executing actions," the boundaries of security must be redrawn accordingly. Staying alert, understanding how these attacks work, and applying structured permission design rather than blindly trusting model self-judgment — that's the most reliable approach for now.