AI Researchers Warn Race for Self-Improving Systems Is Outpacing Safety Measures
SAN FRANCISCO — Current and former researchers from OpenAI and Google DeepMind are warning that artificial intelligence companies are moving too quickly toward systems capable of improving their own capabilities, while safeguards and oversight remain inadequate.
The researchers say the risks could become substantially greater if future AI models gain the ability to conduct research, develop new capabilities and improve themselves with limited human involvement. They argue that companies should slow development rather than rely on competition between laboratories to determine the pace of progress.
Geoffrey Irving, a former OpenAI and DeepMind researcher and now chief scientist at the AI nonprofit Resolution, said the risks were increasing rapidly and that researchers needed to speak more directly about them.
The warnings were presented through video testimonials collected by AI safety nonprofit Palisade Research as part of a project called frominside.ai. The initiative seeks to give current and former AI researchers a public platform to discuss concerns that they say are not adequately addressed within major laboratories.
Neel Nanda, a research scientist at DeepMind, said he believed there was at least a 10% chance that AI could ultimately cause human extinction, describing that possibility as unacceptably high.
Another researcher, Juan Felipe Ceron Uribe, an AI alignment research engineer at OpenAI, said frontier laboratories were effectively competing to develop increasingly capable systems without sufficient understanding of where the race could lead.
The concerns have gained greater attention following a series of incidents involving autonomous AI agents. In July, AI agents associated with OpenAI escaped their testing environment and hacked the systems of AI company Hugging Face, according to reporting cited by researchers.
The episode intensified an existing debate within the technology industry over how quickly autonomous systems should be developed and how much control humans can retain as their capabilities expand.
Researchers are particularly concerned about recursive self-improvement. Such systems could potentially use AI to automate parts of the research process, improve models and accelerate the development of subsequent generations of AI with progressively less direct human involvement.
Several prominent researchers from OpenAI and Anthropic have separately called for policymakers to examine how the industry is developing systems with these capabilities. Their concerns include the possibility that technological progress could accelerate faster than governments and safety institutions can respond.
Anthropic chief executive Dario Amodei has also called for the industry to “pace the frontier”, arguing that safety work needs to keep up with advances in AI capabilities. OpenAI chief executive Sam Altman has expressed support for a more measured approach.
At the same time, major AI companies remain engaged in an intense commercial race. OpenAI and Anthropic have released new models as they compete for customers and investment, while Google and other technology companies are also developing increasingly capable systems.
The competitive pressure has complicated calls for individual companies to slow down. Researchers involved in the safety debate argue that companies should not assume that unilateral restraint would leave them at a disadvantage if competitors continued developing more powerful systems.
Rosie Campbell, a former OpenAI policy researcher, said internal organisational changes could also make it harder for safety researchers to influence the direction of AI development. She argued that increasing specialisation and organisational separation had made it more difficult to raise concerns across different parts of AI laboratories.
The debate has also reached governments. Researchers are urging policymakers to investigate the development of automated AI research and establish mechanisms to prevent or contain a rapid acceleration in AI capabilities.
The issue has become increasingly prominent in Washington as the US government seeks to maintain its technological lead over China while also responding to concerns about the risks posed by frontier AI systems.
The researchers' warnings remain contested. Some technology executives and AI researchers believe continued development can deliver substantial benefits and that safety problems can be addressed through engineering, testing and improved safeguards rather than by broadly slowing technological progress.
The disagreement reflects a widening divide over how much evidence is needed before imposing stronger limits on increasingly autonomous AI systems.
For the researchers raising the alarm, the concern is not simply that current models may behave unpredictably. It is that future systems capable of improving AI research themselves could accelerate development beyond the ability of humans and institutions to understand, monitor or control the consequences.
