AI Existential Risk Probabilities Are Unreliable — So Why Are They Still Used in Policymaking?

When a report claims "the probability of AI causing human extinction is 10%," will policymakers take that number seriously? A growing body of evidence suggests that such seemingly precise probability estimates are actually built on layers of speculation and pseudo-quantification — far from sufficient to support serious public decision-making.

How Speculation Gets "Laundered" into Data

The original article points out bluntly that probability estimates for AI existential risk are undergoing a "laundering" process — subjective, highly uncertain guesses are being packaged as quantified, scientifically reliable data.

This process typically follows a fixed pattern. First, a researcher or institution proposes a number based on intuition, analogy, or a simple model — say, "the probability of superintelligence going out of control is 5%." Next, this number gets cited in subsequent research, where the citers often no longer question the reasonableness of the original assumptions but treat it as a given input for secondary calculations. After several rounds of citation and stacking, the initial rough guess acquires the appearance of "literature support" and eventually appears in policy briefs as a basis for decision-making.

The problem is that each step in this chain loses uncertainty information. The subjective judgments, model limitations, and parameter sensitivities in the original estimate are stripped away layer by layer during transmission, leaving only a bare percentage number.

Why Probability Estimates Are Hard to Make Reliable

Probability estimates for AI existential risk face several structural difficulties that make them fundamentally different from climate models or epidemiological forecasts.

First, the reference class is missing. We have only one civilization and have never experienced the emergence of superintelligence, so we cannot infer baseline probabilities from historical data the way we assess flu vaccine effectiveness. Any number must rely on some form of prior assumption, and those assumptions themselves cannot be empirically verified.

Second, model uncertainty far exceeds parameter uncertainty. Even if we accept a theoretical framework about AI development, the correctness of the framework itself — including the possibility of an intelligence explosion, the difficulty of goal alignment, the capacity for human coordination — remains unresolved. Under different theoretical frameworks, the answer to the same question can differ by orders of magnitude.

Third, probabilistic language itself may not apply. For unique, unprecedented historical events, whether it is even appropriate to talk about "probability" is itself a philosophical question. Some scholars argue that such situations are better discussed through possibility narratives or scenario analysis rather than forcing a numerical value onto them.

The original article emphasizes that these difficulties are not technical details but fundamental obstacles that directly affect whether probability estimates can serve as policy tools. A number that cannot be calibrated or falsified lacks operational meaning in legal or regulatory contexts.

The reference class problem is a classic puzzle in the philosophy of probability. When we say an event has an X% probability, we usually imply an assumption: that the event belongs to some repeatably observable class of events, and we can count the historical frequency of that class. For example, when evaluating the effectiveness of a new drug, we can refer to clinical trial data for similar drugs. But AI existential risk — especially superintelligence going out of control — is a unique event with no precedent, so we cannot construct a statistically meaningful reference class. This means any probability number must rely on prior assumptions that cannot themselves be tested against data.

In probability and statistics, parameter uncertainty and model uncertainty are two different levels of the unknown. Parameter uncertainty refers to: assuming the model structure is correct, but certain parameter values are unknown (e.g., "does AI development speed double every year or every decade?"). This kind of uncertainty can usually be quantified with confidence intervals. Model uncertainty refers to: the model itself may simply be wrong — the theoretical framework we have chosen (e.g., "an intelligence explosion is inevitable" or "goal alignment can be fully solved") may be completely off. In the AI risk domain, model uncertainty far exceeds parameter uncertainty, meaning that even if we can precisely estimate certain parameters, the overall conclusion remains highly unreliable.

Policy Risks of Pseudo-Quantification

Bringing unreliable probability estimates into policy discussions can have consequences more serious than mere "imprecision."

The most direct risk is false consensus. When a number is repeatedly cited within policy circles, it gets treated as "scientific consensus" when in reality it may just be the subjective judgment of a few researchers. This false consensus narrows the space for discussion, making skeptics appear "irrational" or "not serious."

Second is resource misallocation. If policymakers allocate R&D funding or regulatory attention based on an unstable probability estimate, they may over-invest in low-probability scenarios while neglecting more urgent but "less sexy" risks.

A deeper risk is the erosion of decision-making legitimacy. If AI policy is ultimately built on numbers that cannot withstand scrutiny, then once the public or subsequent researchers reveal their fragility, the credibility of the entire policy framework will be damaged. The original article warns that value judgments dressed up as "science" will eventually be exposed under examination.

False consensus in social psychology refers to people's tendency to overestimate the extent to which others agree with their own views. In policy contexts, this effect is amplified: when a number is repeatedly cited in the literature, people in policy circles mistakenly believe it represents a broadly accepted scientific conclusion, when it may actually be the subjective estimate of a few researchers. This false consensus creates an information cocoon, marginalizing voices that question the number and thereby narrowing the diversity and prudence of policy discussion.

How Policymakers Should Treat Probability Numbers

This is not to say that AI risk discussions should exclude all quantitative attempts. The original article's position is more nuanced: probability estimates can serve as thinking tools, but they cannot directly serve as a basis for decision-making.

A pragmatic path is to distinguish between different levels of uncertainty. For problems dominated by parameter uncertainty, range estimates can be given with assumptions clearly labeled; for problems dominated by model uncertainty, scenario narratives and robustness analysis should be prioritized over point probabilities.

Policymakers need to cultivate a kind of "quantitative literacy" — the ability to interrogate a number's sources, assumptions, and sensitivities rather than being dazzled by the precision of decimal places. When a report states "the probability of existential risk is X%," the truly important questions are not what X is, but which assumptions the number is most sensitive to, under what conditions it would change dramatically, and what would happen if it were wrong.

In the field of AI governance, acknowledging the boundaries of our ignorance may be more conducive to responsible decision-making than pretending to have precise answers.

One-line Summary

AI existential risk probability estimates are unreliable because subjective speculation gets laundered into quantitative data, and should not be used directly as a policy basis.

Paragraph Summary

The article points out that probability estimates for AI existential risk are undergoing a "laundering" process: subjective guesses, after repeated citation, acquire the appearance of literature support, while uncertainty information is stripped away layer by layer during transmission, ultimately appearing as precise percentages in policy briefs. However, such estimates face structural difficulties including the missing reference class, the dominance of model uncertainty, and the questionable applicability of probabilistic language — making them fundamentally different from climate models or epidemiological forecasts. Introducing unreliable numbers into policy discussions leads to false consensus, resource misallocation, and erosion of decision-making legitimacy. The article argues that probability estimates can serve as thinking tools but should not directly serve as a basis for decision-making; policymakers should interrogate the sources, assumptions, and sensitivities of numbers, and acknowledging the boundaries of ignorance is more conducive to responsible decisions than pretending to be precise.