1,368 Real-World AI Incidents Reveal an Evaluation Blind Spot: Risk Beyond the Model

AI evaluation techniques are becoming increasingly sophisticated: we measure accuracy, test for hallucinations, run fairness checks, assess robustness, and compare different systems using ever more demanding benchmarks. Yet when researchers systematically examine real-world AI failures, a recurring pattern emerges—the failures that cause the most severe consequences are often not just failures of the model itself.

Failures Beyond the Model: Systematic Patterns Across 1,368 Incidents

A study of 1,368 documented real-world AI incidents found that whether technical problems—whether incomplete data, insufficient predictive performance, or overlooked failure modes—actually cause harm depends heavily on what happens after the model is deployed.

The severity of consequences is closely tied to several key factors: the decision-making authority granted to the AI, whether humans meaningfully intervene, how the system is actually used in business contexts, and whether anyone detects problems early enough.

Data analysis shows a 4.7x increase in correlation between severe monitoring and detection failures and fatal outcomes, while high AI decision-making authority is associated with a 2.8x increase. The researchers emphasize that these are associations observed in reported incidents, not causal estimates, and that the underlying data suffers from reporting bias. Even so, these numbers raise a critical question: are major AI risks shaped not only by how models fail, but also by what the surrounding system allows to happen?

Reporting Bias is a methodological limitation that must be considered when interpreting such research findings. The "incidents" researchers can analyze are, by nature, only those that have been recorded, reported, and entered into public databases. This means two types of bias may coexist: first, underreporting, where many severe failures are never made public due to confidentiality agreements, reputational concerns, or regulatory gaps; second, selective reporting, where media, regulators, or researchers tend to focus on events that caused visible harm while overlooking failures that "happened" to cause no consequences. When researchers claim a "4.7x correlation between monitoring failures and fatal outcomes," this number describes a statistical association within the reported sample, not the true causal effect across all AI incidents. Moreover, incident database entries are often submitted by different sources using different standards, with inconsistent variable definitions—whether a "monitoring failure" gets flagged may itself depend on incident severity: the more severe the incident, the more thoroughly it is investigated, and the more likely monitoring deficiencies are exposed. Therefore, the most valuable contribution of this type of research is not precise risk multipliers, but rather the revelation of a stable structural pattern: the control layers beyond the model (oversight, authority, monitoring, reversibility) play a decisive role in the transmission of harm.

Three Real Cases: The Boundary Between Automation and Human Oversight

Michigan's MiDAS Unemployment System

Michigan's MiDAS unemployment system automatically issued fraud determinations and penalties. The state unemployment agency later reviewed 40,195 fraud determinations decided entirely by computer program—85% were overturned. Additionally, the agency reviewed 22,589 cases initiated by computer and referred to investigators—44% of those determinations were overturned.

The lesson from this case is not simply that "automation makes mistakes"—merely adding a human step did not make the problem disappear either. The more important question to ask is: did the process provide a meaningful opportunity to detect and correct errors before consequences reached real people?

South Australia Aged Care Monitoring Pilot

An independent review found that the monitoring system generated over 12,000 alerts that were not verified as real incidents. In the final months, staff were unable to respond to every alert, and at least one unanswered alert involved a real fall experienced by a resident.

UnitedHealth Class Action Lawsuit

In an ongoing U.S. class action lawsuit, plaintiffs allege that employees were required to operate within 1% of algorithm-predicted care durations. The defendants deny these allegations, the litigation is ongoing, and no liability has been determined. Regardless of how the case is ultimately decided, it points to a broader question: when AI systems make errors, can humans realistically object and change the outcome?

The "right of human objection" in algorithmic decision-making is a concept frequently mentioned in AI governance but rarely rigorously defined. Technically, "humans can override AI" usually means the system includes an override button or approval step; but in organizational practice, genuine objection rights depend on three conditions: information accessibility (can humans obtain the data and explanations needed to make independent judgments), time feasibility (do humans have enough time to complete review before consequences occur), and incentive compatibility (does overriding the AI lead to negative consequences for individual performance, promotion, or workplace relationships). The core allegation in the UnitedHealth case—that employees were required to operate within 1% of algorithm-predicted durations—precisely points to the third condition: even if the system allows manual adjustment, organizational pressure may make "overriding" infeasible in practice. This echoes the South Australia aged care case: staff did not lack the authority to handle alerts; rather, the volume of alerts exceeded the cognitive bandwidth for meaningful human review. This suggests evaluators need to distinguish between nominal oversight and effective oversight: the former requires only drawing a human node on a process diagram, while the latter requires verifying whether humans can substantively change outcomes under real workload, information conditions, and organizational incentives.

Evaluation Needs a Second Question

Traditional evaluation still needs to answer whether a model works under expected conditions—this question remains essential. But real-world incidents show that it cannot be the only question we ask.

We also need to ask: under what conditions will the system fail? And what happens when that failure reaches the real world? This question shifts evaluation toward proactively searching for the conditions that could break through our confidence.

What happens when the underlying data changes over time? What happens when thousands of alerts arrive faster than humans can meaningfully review them? What happens when humans can technically override the AI, but organizational pressure makes such override difficult in practice?

From Model Evaluation to System Evaluation

Model evaluation is not becoming less important—it is becoming one layer within a larger evaluation problem. Researchers frame this through three interconnected questions:

Where might the system break down—across data, design, predictive validity, fairness, explainability, and failure modes? Can such failures reach the real world through excessive authority, weak oversight, limited reversibility, or insufficient monitoring? How significant is such failure in financial, legal, operational, safety, reputational, or human-life terms?

Not every technical weakness deserves equal attention. The goal should not merely be to identify more possible failures, but more importantly to identify the failures that truly matter, understand how they propagate through the system, and determine whether sufficient controls exist to intercept them before they cause actual harm.

This becomes increasingly critical as AI systems shift from recommending what people should do to taking more consequential actions on their own. The next generation of AI evaluation needs better benchmarks and stronger model-level measurement, as well as a broader capability: evaluating the entire system around the model, finding failures before the real world finds them for us.

Reversibility is an often underestimated dimension in system-level AI evaluation, yet it directly determines the magnitude of consequences from failure. Reversibility refers to the extent to which an automated decision or action can be undone, corrected, or compensated before causing irreversible harm. In the Michigan MiDAS case, wrongly determined fraud could be overturned upon review months or even years later, but unemployed individuals may have already lost housing, credit, or job opportunities in the interim—economic penalties may be reversible on paper, yet irreversible in life trajectories. By contrast, the unanswered fall alert in South Australia aged care involved physical safety, where certain consequences are physically irreversible. Evaluating reversibility requires answering several specific questions: Is there a time window between an erroneous decision and its final effect? Is that window actively monitored? Is the revocation process accessible and affordable to those affected? Are there cascading effects where one error triggers multiple downstream automated actions? As AI moves from "recommending" to "acting," the importance of reversibility assessment is rising, because autonomous action compresses the time and space for human intervention and error correction.

One-line Summary

1,368 real-world AI incidents show that the most severe failures occur at the system layer after model deployment, and evaluation must expand from models to monitoring, authority, and reversibility.

Paragraph Summary

A study of 1,368 real-world AI incidents found that the failures causing the most severe consequences are often not in the model itself but in the post-deployment system environment. Data shows that monitoring and detection failures correlate with fatal outcomes at 4.7x, while high AI decision-making authority correlates at 2.8x. Cases including Michigan's MiDAS system where 85% of automated fraud determinations were overturned, over 12,000 unverified alerts in South Australia aged care, and the UnitedHealth algorithm duration lawsuit collectively show that neither fully automated decisions nor simply adding a human step is sufficient—what matters is whether humans meaningfully intervene and can detect and correct errors before consequences occur. The article argues that traditional model evaluation still needs to answer "does the model work," but must add a second question: under what conditions will the system fail, and how does failure reach the real world. AI evaluation is shifting from single-model assessment to system-level evaluation encompassing data, design, authority, oversight, reversibility, and monitoring.