AI Detectives' Trust Crisis: How Conflict-Aware Evidence Verification Saves Cybersecurity Investigations

When AI Detectives Stop Trusting Their Tools: An Overlooked Security Blind Spot

Imagine you're a cybersecurity analyst facing a flood of alerts, relying on multiple detection tools to make judgments. These tools may come from different vendors, use different algorithms, and may even be poisoned by attackers. When they disagree, who do you trust?

This is exactly the real-world dilemma LLM agents face in security operations. The latest research, When AI Investigators Cannot Trust Their Tools: Conflict-Aware Evidence Verification for Cyber Threat Investigation, systematically examines for the first time how conflicts between tools affect AI investigators. The study points out that existing LLM agents, when combining outputs from multiple detection tools, often default to assuming these tools are trustworthy—overlooking the fact that machine-generated evidence may conflict or be wrong.

The Fragile Undefended Investigator: One Poisoned Tool Is Enough to Bring It Down

On a corrected CIC-IDS2017 dataset, the research team used four heterogeneous detectors to generate labels and confidence scores, and built a threat model varying the number of poisoned tools, the persistence of poisoning, and the confidence of erroneous outputs. They tested open-source LLMs ranging from 3B to 14B parameters.

The results are alarming: unverified investigators are extremely fragile. With just one of four detectors poisoned, naive accuracy drops to 0.50–0.72; with three poisoned, all unverified investigators fall to 0.13 or below. More subtly, even without malicious poisoning, honest disagreement alone is enough to make them abstain on up to 63% of traffic.

This reveals a deeper problem: LLM agents lack the ability to independently judge evidence quality. They rely excessively on tool reports, while the tools themselves may be wrong or manipulated.

CIC-IDS2017 is an intrusion detection dataset released by the Canadian Institute for Cybersecurity (CIC), containing real network traffic over 5 days, annotated with various attack types such as DDoS, brute force, and web attacks. Because it closely reflects real-world scenarios, it is widely used for IDS evaluation, but it also has known issues such as labeling errors and imbalanced traffic distribution. The research team used a "corrected" version precisely to reduce the interference of these noises on the conclusions.

In the threat model, "tool poisoning" refers to an attacker tampering with a detector's output so that it assigns a "benign" label or an erroneous high-confidence judgment to malicious traffic. Poisoning persistence describes whether such tampering is sporadic or continuous; the confidence of erroneous outputs determines how much weight the erroneous evidence will be given during fusion. Together, these three dimensions characterize the capability and cost an attacker needs to manipulate an AI investigator, and also form a stress-testing framework for evaluating LLM robustness.

Conflict-Aware Evidence Verification (CAV): From Trusting Tools to Trusting Checks

Faced with this dilemma, the researchers propose the Conflict-Aware Evidence Verification (CAV) framework. It consists of three established components:

• Equal-weight voting: ignores tool-reported confidence to avoid being misled by high-confidence erroneous outputs.
• Evidence conflict gating: detects whether there is significant conflict among tools.
• Independent nearest-neighbor trust check: independently verifies leading candidates without relying on the tools' own judgments.

The CAV-LLM prompt was frozen before testing to ensure evaluation fairness. Under the harsh condition of three poisoned detectors, CAV-LLM achieves 0.84–0.88 accuracy, while the naive investigator reaches at most 0.08. At the same time, it invokes the LLM on only 0.4% of clean traffic, greatly reducing computational overhead.

The study also finds that when most tools are unpoisoned, a fixed verification rule (not relying on the LLM) is equally or even more accurate; only when the majority of tools are poisoned does LLM intervention show its advantage.

Equal-Weight Voting is a classic ensemble fusion strategy that ignores the self-reported confidence of each base learner and assigns equal weight to each vote. The benefit is avoiding "overconfident errors" dominating the result—when a tool is poisoned and outputs a high-confidence erroneous label, weighted voting amplifies its influence, whereas equal-weight voting limits it to the same number of votes as other tools. The cost is losing information when tool reliability varies greatly.

Evidence Conflict Gating draws on the idea of conflict measurement from evidence theory: when the decision distributions of multiple sources are highly inconsistent, the system triggers an additional verification process instead of forcibly fusing them. The Independent Nearest-Neighbor Trust Check, meanwhile, reviews leading candidates using an independent feature space or reference samples without relying on the tools' own judgments—equivalent to equipping the AI investigator with a "disobedient third-party auditor." The core logic of combining the three is: first reduce reliance on confidence, then identify conflicts, and finally adjudicate with an independent channel.

Noise in the Independent Channel: CAV's Limitations and Trade-offs

CAV's core lies in the independent trust check—but what if this independent channel itself is noisy? The study further simulates the case where the independent channel makes errors.

The results show that CAV inherits errors from the independent channel: when two or more tools are poisoned, CAV still maintains the lead; but when at most one tool is poisoned, treating the independent channel as an additional voter is actually better. This means the verification mechanism itself also needs to be measured and protected—trust has shifted from the tools to the check, and the check is not absolutely reliable.

This finding has important implications for real-world deployment: there is no one-size-fits-all solution. Security teams need to dynamically adjust verification strategies based on the threat environment and continuously monitor the reliability of the verification channel.

Noise in the independent channel can be understood as the false positive rate and false negative rate of the verification mechanism itself. In security operations, the independent channel may be a rule-based detector, a sandbox in an isolated network, or a manually maintained threat intelligence database—all of which may also err due to outdated rules, sandbox escapes, or intelligence poisoning. By simulating this situation, the study is effectively asking: when the "referee" may also misjudge, how does the reliability of the entire verification architecture decay?

This involves a more general system design principle—the verifier's paradox: any verification mechanism needs to be verified itself; otherwise it merely shifts the trust problem up one level. In practice, this means CAV-like frameworks need accompanying monitoring metrics (such as the agreement rate between the independent channel and the tools' majority opinion, and abstention-rate drift) and fallback strategies (such as reverting to fixed-rule voting when the independent channel's confidence falls below a threshold), rather than treating the independent check as an absolutely trustworthy baseline.

Multi-Source Corroboration and Industry Implications

According to analyses by Bilibili creator "AI Security Observer" and the YouTube channel "SecAI," this research echoes the industry's attention to the "tool trust" problem in AI security. Both independent sources point out that as LLM agents become widespread in security operations centers (SOCs), ensuring the robustness of their decisions will become a key challenge.

The research consensus is that simple tool integration is insufficient to guarantee security; independent verification mechanisms must be introduced. The disagreement lies in the choice of verification strategy—fixed rules and LLM verification each have applicable scenarios depending on the degree of poisoning. A unique insight from a single source is that the CAV framework can be generalized to other multi-tool collaborative AI systems, such as medical diagnosis or financial risk control.

Conclusion: The Shift of Trust and the Attribution of Responsibility

This research reveals a fundamental shift: in the world of AI investigators, trust cannot rest on tools but must be built on verifiable checks. CAV offers a feasible path, but it also reminds us that the verification mechanism itself needs careful design and continuous protection. For developers building secure AI systems, this is not just a technical issue but also a question of responsibility attribution.

As AI agents penetrate deeper into critical domains, how to balance automation and verification will become the next frontier.

One-line Summary

LLM security investigators over-trust detection tools; the CAV framework uses independent verification to raise accuracy under poisoning from 0.08 to 0.88.

Paragraph Summary

This article interprets a study on the tool trust problem of LLM agents in cybersecurity investigations. Using a corrected CIC-IDS2017 dataset, the study constructs multi-detector conflict and poisoning scenarios and finds that unverified LLM investigators are extremely fragile: with just one tool poisoned, accuracy drops to 0.50–0.72; with three poisoned, it falls below 0.13, and honest disagreement alone leads to up to 63% abstention. To address this, the authors propose the Conflict-Aware Evidence Verification (CAV) framework, which through equal-weight voting, evidence conflict gating, and an independent nearest-neighbor trust check achieves 0.84–0.88 accuracy when three tools are poisoned, while invoking the LLM on only 0.4% of clean traffic. The study further points out that noise in the independent verification channel causes CAV to inherit errors, requiring dynamic strategy adjustment based on poisoning degree and shifting trust from tools to verifiable checks.