False Positive
One of the core metrics in scam detection, referring to the case in which a legitimate message is incorrectly judged as a scam. An excessively high rate causes users to miss important information.
Core Facts
If the judgment criteria of an AI reporting mechanism are biased, it may generate a large number of false positives and could even be maliciously exploited to attack competing systems.
Text generated by large language models tends to have low perplexity, with word choices that are overly consistent with probabilistic expectations, lacking the unexpectedness of human writing.
Timeline (last 90 days)
Matt Pocock's core judgment is: making the review "super ambitious" and proposing more structural fixes inevitably increases false positives, but false positives are easy to handle—what is truly dangerous is false negatives.
For equipment where downtime costs are extremely high, more false positives should be tolerated in exchange for few false negatives; for general equipment, a slightly higher false negative rate can be accepted to control inspection workload.
A false positive means the system issues an alert but the equipment is actually normal; a false negative means the equipment does fail but the system does not alert.
In learning products aimed at children, the cost of false positives is equally high: a filter that treats a normal English reading comprehension question as privacy phishing will directly damage the product experience.
A false positive means the scanner or tool reports a vulnerability that does not actually exist, while a false negative means a real vulnerability is not detected.
In Agent security, a false negative means a real attack is let through, which can lead to irreversible consequences such as key exfiltration; a false positive merely blocks a legitimate request, which can be mitigated by retrying or manual review. The costs of the two are asymmetric.
When a port scan detects that the target has port 3306 (MySQL's default port) open, it is flagged as a risk; however, the port may actually be unreachable from the external network due to access control whitelists, WAF, or cloud security group rules, resulting in a false positive.
If the judgment criteria of an AI reporting mechanism are biased, it may generate a large number of false positives and could even be maliciously exploited to attack competing systems.
If educational institutions determine academic misconduct based solely on detector results, it may cause false positives (human-original work misidentified as AI) and false negatives (AI content that passes after rewriting).
All Facts (10)
If the judgment criteria of an AI reporting mechanism are biased, it may generate a large number of false positives and could even be maliciously exploited to attack competing systems.
65%VerifiedText generated by large language models tends to have low perplexity, with word choices that are overly consistent with probabilistic expectations, lacking the unexpectedness of human writing.
65%UnverifiedMatt Pocock's core judgment is: making the review "super ambitious" and proposing more structural fixes inevitably increases false positives, but false positives are easy to handle—what is truly dangerous is false negatives.
50%UnverifiedA false positive means the system issues an alert but the equipment is actually normal; a false negative means the equipment does fail but the system does not alert.
50%UnverifiedFor equipment where downtime costs are extremely high, more false positives should be tolerated in exchange for few false negatives; for general equipment, a slightly higher false negative rate can be accepted to control inspection workload.
50%UnverifiedIn learning products aimed at children, the cost of false positives is equally high: a filter that treats a normal English reading comprehension question as privacy phishing will directly damage the product experience.
50%UnverifiedA false positive means the scanner or tool reports a vulnerability that does not actually exist, while a false negative means a real vulnerability is not detected.
50%UnverifiedIn Agent security, a false negative means a real attack is let through, which can lead to irreversible consequences such as key exfiltration; a false positive merely blocks a legitimate request, which can be mitigated by retrying or manual review. The costs of the two are asymmetric.
50%UnverifiedWhen a port scan detects that the target has port 3306 (MySQL's default port) open, it is flagged as a risk; however, the port may actually be unreachable from the external network due to access control whitelists, WAF, or cloud security group rules, resulting in a false positive.
50%UnverifiedIf educational institutions determine academic misconduct based solely on detector results, it may cause false positives (human-original work misidentified as AI) and false negatives (AI content that passes after rewriting).
50%