Anthropic Proposes AI Safety Bill: Regulation Only Triggered by 50 Deaths or $1 Billion in Losses

Event Overview

According to discussions in the Hacker News community, AI company Anthropic has proposed a draft AI safety bill whose core trigger threshold is set as: causing 50 deaths or $1 billion in economic losses before the corresponding regulatory or liability mechanisms are activated. This design quickly drew attention and controversy in the tech community—supporters argue it leaves breathing room for innovation, while critics worry the threshold is too high and effectively toothless.

The Logic and Controversy of the Threshold Design

As a frontier AI lab, Anthropic has consistently advocated "responsible scaling." From a legislative-technique perspective, setting quantitative trigger conditions is nothing new—aviation, pharmaceuticals, and other fields have similar thresholds. But the figure of 50 deaths looks especially jarring when applied to software systems: casualties from traditional software incidents are usually far below this, while the risk of AI systems lies precisely in their hard-to-predict, large-scale impact.

The controversy centers on two points. First, the $1 billion loss threshold imposes almost no constraint on large tech companies, yet could be catastrophic for startups—an inherent asymmetry. Second, triggering after the fact means regulation happens after harm has already occurred rather than preventing it beforehand, which creates tension with the original intent of "safety" legislation.

Responsible Scaling is an internal governance approach proposed by frontier labs such as Anthropic. Its core is to predefine safety evaluation and deployment conditions according to model capability levels: the stronger the capability, the stricter the required testing, monitoring, and risk mitigation measures. It borrows the "tiered licensing" logic from aviation and nuclear energy, emphasizing the identification of dangerous capabilities (such as assisting with biological or cyber attacks) before training and release, and deciding on that basis whether to continue scaling. This framework is corporate self-regulation in nature and lacks external enforcement, which is why Anthropic turned to a legislative proposal, attempting to translate internal standards into public rules. The controversy lies in whether the capability thresholds and liability thresholds set by companies themselves are strict enough, and who verifies that a model has indeed reached a certain level.

Community Reaction: Support and Skepticism Coexist

Among the 17 comments on Hacker News, the discussion showed clear divisions. Some developers endorsed the phased-regulation approach, believing that imposing strict compliance too early would stifle the open-source ecosystem and small teams; other voices bluntly called it "a liability exemption clause tailor-made by big companies for themselves," pointing out that Anthropic is both a rule advocate and a potential beneficiary, creating a conflict of interest.

Such discussions reflect a deeper question: when AI companies proactively propose regulatory frameworks, how do we ensure the rule design is not captured? There is currently no mature answer.

Regulatory capture refers to the phenomenon in which a regulated industry, through lobbying, standard-setting, or personnel turnover, causes regulators or the rules themselves to favor its commercial interests. In AI governance, this risk manifests as: leading labs, relying on their technical and compliance resources, dominate rule drafting and set thresholds at levels only they can bear, thereby raising entry costs for latecomers. Common industry responses include conflict-of-interest disclosure, independent multi-stakeholder standards bodies, and tiered obligations applicable to open-source and small-to-medium teams. But to date, the AI field still lacks an independent review mechanism comparable to financial or pharmaceutical regulation, which is the structural reason the community remains wary of Anthropic's proposal.

Comparison and Industry Impact

Compared with the risk-tiering approach of the EU AI Act, Anthropic's plan is closer to a "harm-threshold trigger" model: the former emphasizes ex-ante compliance assessment, while the latter focuses on ex-post accountability. Both paths have pros and cons, but they share a challenge: the causal chain of AI harm is complex, and whether an event like 50 deaths can be clearly traced to a single model or deployer is doubtful in practice.

You may not have noticed that the draft is currently only at the proposal stage, with no legislative process backing it. Its actual impact is more at the level of public opinion—it has pushed the question of "how high should the AI regulatory threshold be" back into the public eye.

The EU AI Act adopts horizontal risk-based tiering: applications with unacceptable risk are outright prohibited; high-risk systems must fulfill ex-ante compliance assessment, data governance, technical documentation, human oversight, and other obligations; general-purpose large models have separate transparency and systemic risk assessment requirements. Its regulatory object is "purpose and scenario," not "harm outcome." Anthropic-style harm-threshold triggering is an ex-post liability path, in which accountability is only initiated after specific-scale casualties or economic losses occur. The key difference between the two lies in evidence and attribution: ex-ante compliance only requires proving that processes are in place, while ex-post accountability must prove a causal chain between model outputs and specific harm—and multi-party deployment, fine-tuning, and prompt engineering of AI systems make this chain extremely difficult to close.

Conclusion

Anthropic's draft bill exposes a fundamental dilemma in AI safety governance: regulation must prevent catastrophic risk without stifling innovation. Whether the threshold of 50 deaths or $1 billion in losses is reasonable depends on how we weigh prevention costs against potential harm. This discussion has only just begun.