Pretraining Poisoning: A New Threat of Backdoors in AI Agents

On Hacker News, a discussion titled "Agentic Backdoors via Pretraining Poisoning" drew attention. The post points out that by injecting malicious data into a model during the pretraining phase, attackers can implant a backdoor targeting AI agents (Agentic AI)—a backdoor that lies dormant normally but may be triggered once the agent is given the ability to execute tasks, creating security risks.

Pretraining Poisoning: More Than Just Data Contamination

Traditional backdoor attacks mostly occur during the fine-tuning phase, where a small number of malicious samples cause the model to produce incorrect outputs under specific inputs. Pretraining poisoning is more stealthy: malicious data is absorbed while the model learns general language capabilities, and the backdoor may be deeply embedded in the model parameters, making it difficult to detect through conventional methods.

This attack is especially dangerous for agents. Agents must not only understand language but also call tools, execute operations, and make decisions. Once the backdoor is triggered, attackers may remotely control the agent to perform unauthorized operations, such as leaking sensitive information, executing malicious code, or damaging systems.

The difference between pretraining and fine-tuning is key to understanding this attack. Pretraining refers to the process in which a model learns general language representations on large-scale unlabeled corpora, typically requiring massive data and enormous compute, producing foundation models (such as the GPT series) with broad language understanding capabilities. Fine-tuning further trains a pretrained model on small-scale labeled data for specific tasks to adapt it to a particular application. Traditional backdoor attacks mostly target the fine-tuning phase because attackers can control the fine-tuning data. Pretraining poisoning occurs further upstream: after malicious data is mixed into the pretraining corpus, the backdoor pattern is encoded into the model's underlying parameters, affecting all downstream tasks. The stealthiness of this attack lies in the fact that pretraining data is enormous (often trillions of tokens), a single malicious sample accounts for an extremely low proportion, and conventional data review can hardly detect it; at the same time, the backdoor is deeply fused with the model's language capabilities, making it difficult to distinguish normal parameters from malicious ones during detection.

Unique Risks Facing Agents

Unlike ordinary chatbots, agents possess the ability to "act." They can access APIs, read and write files, send emails, and even control physical devices. This means that once a backdoor is activated, the destructive power escalates from "saying the wrong thing" to "doing the wrong thing."

More troublesome is that an agent's decision chain is often long, and the backdoor may be triggered only under specific task sequences or environmental conditions, increasing detection difficulty. In addition, agents typically need continuous learning or adaptation to new environments, which provides attackers with more injection opportunities.

Agentic AI refers to AI systems that can autonomously perceive the environment, plan tasks, call tools, and execute actions, typically represented by AutoGPT, BabyAGI, and various large-language-model-based agent frameworks. Unlike chatbots that only generate text, agents have the ability to "act": they can call external APIs, operate databases, send emails, control physical devices, and more. This capability relies on the "tool calling" mechanism—the model selects and invokes predefined tool functions according to task requirements. An agent's decision-making usually consists of a "plan-execute-observe-reflect" loop, with a long chain that may involve multi-step reasoning. It is precisely this open behavior space and long decision chain that allow backdoor trigger conditions to be designed very specifically (such as specific task sequences or environmental states), thereby bypassing conventional detection. At the same time, agents often need continuous learning or online adaptation, which gives attackers more opportunities to inject malicious data.

Defense Approaches and Open Questions

There is currently no perfect solution against pretraining poisoning. Some research directions include: data provenance and cleaning, training process monitoring, model behavior anomaly detection, and designing least-privilege principles for agents. But challenges remain—pretraining data is massive, and full cleaning is extremely costly; the behavior space of agents is open, and anomaly detection can hardly cover all scenarios.

The discussion on Hacker News also reflects the community's concern about such risks. As agent applications accelerate toward deployment, how to ensure their supply chain security—from pretraining data to the deployment environment—will become a question that must be answered.

Data provenance and cleaning is the first line of defense against pretraining poisoning. Data provenance refers to recording the source, processing, and transformation history of training data so that contamination sources can be traced when anomalies are found. Cleaning includes rule-based or model-based filtering to remove suspicious content. However, pretraining data is massive (often at the TB scale), full manual review is unrealistic, and automated cleaning may miss carefully disguised malicious samples. Training process monitoring aims to detect anomalous behavior during training, such as sudden changes in loss curves or abnormal gradients, but backdoor implantation is often difficult to distinguish from normal training. Model behavior anomaly detection discovers backdoors by testing the output distribution of the model under specific inputs, but the behavior space of agents is open, making it difficult to exhaust all scenarios. The least-privilege principle requires that agents have only the minimum permissions needed to complete tasks, so that even if a backdoor is triggered, the scope of damage can be limited. These defense methods each have limitations and require multi-layered coordination to improve overall security.

Conclusion

Pretraining poisoning brings a new type of backdoor threat to AI agents, and its stealthiness and potential destructive power deserve vigilance. While pursuing agent capabilities, the security community needs to build defense systems in parallel to prevent agents from becoming attackers' "insiders."