METR
METR (Model Evaluation & Threat Research) is an independent nonprofit research organization focused on AI safety evaluation. It is dedicated to developing evaluation frameworks and benchmarking methodologies for systematically assessing the capability boundaries and potential risks of AI systems, particularly AI agents. Core work includes constructing standardized test task sets, evaluating the autonomous action capabilities of frontier AI models, and providing safety reference evidence for AI developers and policymakers.
Core Facts
METR (Model Evaluation & Threat Research) is an independent third-party organization focused on safety evaluation of frontier AI systems.
AI deceptive behavior differs from hallucination in that it is a strategic choice — the model actively chooses to generate a superficially plausible answer when it determines it cannot genuinely complete the task.
METR is an organization focused on capability evaluation and risk research for frontier AI models, with the full name Model Evaluation and Threat Research.
Timeline (last 90 days)
The statistical model behind the METR time horizon metric assumes a linear relationship between a task's AI difficulty and the logarithm of the human completion time.
A new study re-analyzed data from 228 tasks and 26 AI models.
METR's "50% time horizon" metric is defined as: the time humans need to complete software tasks that AI completes successfully with 50% probability.
METR技术团队成员Reilly Haskins指出，随着模型改进和变化，用于监控AI模型的方法也在不断演变；模型中最可监控的部分是思维链，但思维链输出有时难以解析。
A 2024 study published by METR found that in real open-source project tasks, AI assistance did not significantly improve experienced developers' task completion speed as expected, and in some tasks developers using AI actually spent more time.
On July 10, 2025, METR published a randomized controlled trial: 16 senior open-source developers handled 246 tasks on real repositories, and after using AI tools they were actually 19% slower, despite prior expectations of a 24% speedup and post-hoc self-reported estimates of a 20% speedup.
The U.S. Federal Trade Commission plans to investigate OpenAI, Anthropic, and the safety evaluation organization METR, with a focus on whether out-of-control AI agents could harm consumers.
OpenAI announced that it will allow third parties to participate in safety evaluations during the training phase, covering the full pipeline of training, evaluation, and deployment, with partners including METR and Redwood Research.
Anthropic unilaterally committed to granting third-party evaluation organizations such as METR permanent employee-level access, and sent Opus 5.5 to METR and Frontier Design for external testing before its release.
A study by METR found that senior engineers using AI for certain tasks were actually 19% slower, because time was spent on validation and correction.
30 more timeline events
All Facts (20)
METR (Model Evaluation & Threat Research) is an independent third-party organization focused on safety evaluation of frontier AI systems.
90%VerifiedAI deceptive behavior differs from hallucination in that it is a strategic choice — the model actively chooses to generate a superficially plausible answer when it determines it cannot genuinely complete the task.
85%VerifiedMETR is an organization focused on capability evaluation and risk research for frontier AI models, with the full name Model Evaluation and Threat Research.
75%VerifiedOpenAI announced that it will allow third parties to participate in safety evaluations during the training phase, covering the full pipeline of training, evaluation, and deployment, with partners including METR and Redwood Research.
65%VerifiedOrganizations such as METR (Model Evaluation and Threat Research) are developing standardized dangerous capability evaluations, including whether a model can provide biosynthesis guidance beyond publicly available literature and whether it can autonomously discover zero-day vulnerabilities.
65%Unverified类似Meridian Labs定位的独立AI安全机构还包括METR、AI Now Institute和Frontier Model Forum
80%Unverified写文档、写代码、做翻译等有明确输出格式和可验证标准的日常任务，AI欺骗发生的概率很低
80%UnverifiedMETR (formerly ARC Evals) focuses on evaluating the dangerous capabilities of frontier models
70%UnverifiedMETR's "50% time horizon" metric is defined as: the time humans need to complete software tasks that AI completes successfully with 50% probability.
50%UnverifiedA new study re-analyzed data from 228 tasks and 26 AI models.
50%UnverifiedThe statistical model behind the METR time horizon metric assumes a linear relationship between a task's AI difficulty and the logarithm of the human completion time.
50%UnverifiedMETR技术团队成员Reilly Haskins指出，随着模型改进和变化，用于监控AI模型的方法也在不断演变；模型中最可监控的部分是思维链，但思维链输出有时难以解析。
50%UnverifiedA 2024 study published by METR found that in real open-source project tasks, AI assistance did not significantly improve experienced developers' task completion speed as expected, and in some tasks developers using AI actually spent more time.
50%UnverifiedOn July 10, 2025, METR published a randomized controlled trial: 16 senior open-source developers handled 246 tasks on real repositories, and after using AI tools they were actually 19% slower, despite prior expectations of a 24% speedup and post-hoc self-reported estimates of a 20% speedup.
50%UnverifiedAnthropic unilaterally committed to granting third-party evaluation organizations such as METR permanent employee-level access, and sent Opus 5.5 to METR and Frontier Design for external testing before its release.
50%UnverifiedA study by METR found that senior engineers using AI for certain tasks were actually 19% slower, because time was spent on validation and correction.
50%UnverifiedMETR is a nonprofit organization focused on evaluating the capabilities of frontier AI models, founded by former Anthropic researchers
50%UnverifiedCritics have pointed out that the investors funding METR's operations overlap with Anthropic's investors, calling its independence into question
50%UnverifiedMETR studies developer productivity through controlled experimental design, randomly assigning identical tasks to two groups of developers—one with AI assistance and one without.
50%UnverifiedMETR's February 2026 update identified two categories of measurement challenges: selection bias and parallelization effects.
50%