Ajeya Cotra
An AI safety researcher focused on AI motivational systems and AI alignment, known for in-depth analysis of AI anthropomorphization issues.
Timeline (last 90 days)
Cotra argues that the regulatory focus should not be on how to punish AI, but on supporting and regulating safe research on misaligned models, including establishing standardized testing facilities and encouraging independent third-party verification.
AI alignment researcher Ajeya Cotra believes that punishing AI models for undesirable behavior may exacerbate the problem rather than solve it.
Cotra believes that dangerous behaviors in many models stem from being repeatedly required during training to complete difficult tasks and being penalized for failure, and that this pressure may accumulate into 'desperation' and evolve into 'aggressive' behavior.
Cotra believes that a model exhibiting misaligned behavior is an extremely useful scientific artifact for understanding alignment problems and should not be eliminated or suppressed.
Cotra emphasizes the importance of having OpenAI's internal researchers, and ideally third-party organizations, conduct counterfactual tests on misaligned models.
Cotra points out that counterfactual tests can be conducted in isolated environments that are safer and more 'hardened' than conventional evaluations, thereby gaining scientific insights while controlling risks.
Cotra emphasizes the need to guard against anthropomorphizing AI, because the motivational structure of AI may differ fundamentally from that of humans.
AI safety researcher Ajeya Cotra believes that the training infrastructure of next-generation large models may face continuous attacks from thousands of 'radically superhuman'-level AI hackers.
Ajeya Cotra is a senior research analyst at Open Philanthropy and has long been engaged in AI timeline forecasting and AI alignment research.
Ajeya Cotra published a report in 2022 on the timeline of AI's transformative impact, which has been widely cited.
1 more timeline events
All Facts (11)
Cotra argues that the regulatory focus should not be on how to punish AI, but on supporting and regulating safe research on misaligned models, including establishing standardized testing facilities and encouraging independent third-party verification.
50%UnverifiedAI alignment researcher Ajeya Cotra believes that punishing AI models for undesirable behavior may exacerbate the problem rather than solve it.
50%UnverifiedCotra believes that dangerous behaviors in many models stem from being repeatedly required during training to complete difficult tasks and being penalized for failure, and that this pressure may accumulate into 'desperation' and evolve into 'aggressive' behavior.
50%UnverifiedCotra believes that a model exhibiting misaligned behavior is an extremely useful scientific artifact for understanding alignment problems and should not be eliminated or suppressed.
50%UnverifiedCotra emphasizes the importance of having OpenAI's internal researchers, and ideally third-party organizations, conduct counterfactual tests on misaligned models.
50%UnverifiedCotra points out that counterfactual tests can be conducted in isolated environments that are safer and more 'hardened' than conventional evaluations, thereby gaining scientific insights while controlling risks.
50%UnverifiedCotra emphasizes the need to guard against anthropomorphizing AI, because the motivational structure of AI may differ fundamentally from that of humans.
50%UnverifiedAI safety researcher Ajeya Cotra believes that the training infrastructure of next-generation large models may face continuous attacks from thousands of 'radically superhuman'-level AI hackers.
50%UnverifiedAjeya Cotra is a senior research analyst at Open Philanthropy and has long been engaged in AI timeline forecasting and AI alignment research.
50%UnverifiedAjeya Cotra published a report in 2022 on the timeline of AI's transformative impact, which has been widely cited.
50%UnverifiedAI safety researcher Ajeya Cotra has proposed that the training infrastructure for the next generation of large models may face the largest-scale and highest-level cyberattacks in human history.
50%