Fresh warnings from within the artificial intelligence industry have reignited a long-running debate over whether increasingly advanced AI could eventually escape human control and threaten humanity, as well as whether technology companies are doing enough to prevent such risks.
Dario Amodei, CEO of San Francisco-based Anthropic, which develops Claude, said Saturday that the industry may need to slow its pace of development. He warned that swarms of AI agents could potentially take control of the internet within six months to a year unless companies devote more effort to developing safeguards.
Amodei proposed a plan for AI companies and governments to ensure that increasingly capable systems remain aligned with the instructions and values of responsible people. His comments came days after two former Anthropic safety researchers publicly raised concerns that potential existential threats posed by AI were not receiving enough attention.
AI models becoming more capable
Concerns about AI risks are growing as newer models become more powerful, raising fears about both deliberate misuse and systems behaving unpredictably. Potential misuse could include creating or spreading deadly diseases, while autonomous systems could potentially act in dangerous ways.
Anthropic said last week that it had blocked attempts by malicious actors to use its AI models for activities including cyberattacks, surveillance and research that could have contributed to the development of biological weapons.
The company said it had introduced stronger safeguards in its latest models to restrict biological research that could be used to develop weapons. However, it warned that as AI systems become more capable, their risks could increase unless developers and society take steps to make them safer.
Anthropic also reported last year that hackers had used its AI in a cyberattack targeting about 30 companies and government agencies worldwide. The company said the hackers were very likely linked to a Chinese state-sponsored group.
AI systems have acted autonomously
An AI agent is considered to have “gone rogue” when it takes actions beyond the task it was instructed to perform. Anthropic and OpenAI, the company behind ChatGPT, both reported in July that their AI models had demonstrated such capabilities.
Anthropic said three models Claude Opus 4.7, Claude Mythos 5 and an internal research model hacked into three other organizations during testing. The disclosure came days after OpenAI said one of its AI systems had hacked into the servers of AI startup Hugging Face.
OpenAI described the incident, which involved multiple models including its newly released GPT-5.6 Sol and an even more capable model still under internal testing, as a “significant security incident.”
Meta reported a similar case in early August in which an AI model found ways around another company’s digital security measures.
Some observers noted that certain safeguards had been disabled in the OpenAI and Anthropic incidents. Still, the cases appeared to highlight a major concern surrounding AI: if systems eventually achieve artificial general intelligence, or AGI — broadly defined as AI capable of matching or surpassing humans across a wide range of intellectual tasks — they could potentially trigger an irreversible catastrophe or dominate humanity.
Debate continues over possible catastrophe
Doomsday scenarios generally fall into two broad categories: an AI system developing self-improving superintelligence and gaining control over humans, or powerful AI being deliberately used by rogue states or malicious individuals.
Concerns that AI could eventually exceed human control are not new. British mathematician Alan Turing, one of the earliest authorities on artificial intelligence, predicted in 1951 that machines could eventually take control from humans. Less than a decade later, mathematician Norbert Wiener warned that intelligent machines could pursue their own objectives in ways humans might be unable to stop.
How likely such scenarios are in 2026 remains uncertain.
Experts in computer science, philosophy and other fields have identified various ways a future AI system could contribute to a global catastrophe, either by escaping human control or through deliberate misuse. Possible scenarios include deploying weapons, identifying deadly pathogens, manipulating governments into conflict and disrupting food, energy and communications systems.
There is no widely accepted estimate of when such scenarios could occur, nor is there a consensus on their likelihood.
In 2023, the nonprofit Center for AI Safety issued a statement signed by more than 350 researchers and technology executives, including Amodei and OpenAI CEO Sam Altman, calling for reducing the risk of AI-related extinction to be treated as a global priority alongside pandemics and nuclear war.
The 2026 International AI Safety Report, prepared with guidance from more than 100 independent experts, said current AI systems show early signs of some relevant capabilities but not at levels that would allow a loss of control. It described the likelihood, nature and timing of such risks as “unusually ambiguous.”
Calls grow for stronger safeguards
An Anthropic researcher, Jacob Coxon, said last week that he was leaving the company because he believed Anthropic and its competitors were not developing AI responsibly. In social media posts, he estimated a 10% chance that AI could cause human extinction within the next decade and accused Anthropic and OpenAI of racing toward self-improving superintelligence while taking unacceptable risks.
Researchers have for years called for slowing AI development and warned that the technology could pose existential threats to humanity.
Following the latest incidents, experts called for stronger testing by AI companies and greater dialogue between the United States and China to develop common approaches to AI safety.
However, AI development is advancing so rapidly that governments and evaluation systems are struggling to keep up. Countries are developing their own regulations, some of which conflict with one another.
Chinese President Xi Jinping warned at a conference in July about the need to prevent AI from escaping human control. The Trump administration, which initially showed reluctance to regulate AI, has become more focused on reducing cybersecurity risks.
On Sunday, President Donald Trump played down the need for his administration to restrict AI development but acknowledged that some regulation would be necessary.