Researchers play down AI apocalypse fears, citing hurdles to creating a killer virus

Some researchers say one of the most extreme versions of that scenario is overblown.

The debate has reached a fever pitch in the week since Jacob Coxon, a 27-year-old researcher, quit Anthropic and warned that it and his other former employer, OpenAI, were “gambling with our lives” as they raced toward self-improving superintelligence.

Evan Hubinger, who leads Anthropic’s alignment stress-testing team, backed Coxon, saying he personally believes there is a greater-than-10% chance AI could kill all humans within the next decade.

But other researchers have pointed to the gap between what AI could do in the digital world and what it would require to cause real-world harm.

They gave the example of a lethal biological virus created by AI, one of the darkest scenarios that was also cited as a reason to pace AI development by Anthropic CEO Dario Amodei in an essay this month.

AI could design the virus on a computer, but it would be limited by its inability to manufacture it in the real world, the researchers said.

Anselm Levskaya, a staff research engineer at Google who says he has built DNA synthesizers and engineered viruses, wrote on X on Monday that a viral sequence would still have to be physically assembled in a lab and would require a system capable of producing virus particles.

It would then require extensive selection and testing to determine whether it can replicate, spread, evade immune defenses, and remain dangerous — properties Levskaya said depend on complex biological processes that he said cannot simply be designed from a computer.

Eric Xing, the president of Mohamed bin Zayed University of AI and a computer-science professor at Carnegie Mellon University, put it more simply: generating a virus blueprint and producing a real virus are “completely different things.” The gap is in “the material, the manufacturing, and actual biological viability,” he wrote on X on Monday.

Treating the two as equivalent, he added, “is either an intentional attention-harnessing lie or true ignorance.” Researchers said the same logic extends beyond biology.

To turn a plan into real-world harm, an AI would need access to physical equipment, money, computer systems, or people. “The most powerful thing in the world is still constrained by who holds it and how,” Oleksandr Yaremchuk, the CTO and cofounder of Manifold Security, told Business Insider.

Some researchers also drew a line between AI systems carrying out harmful tasks set by people and an AI independently taking actions that harm people.

They said that much of the harm linked to current systems begins with human instruction.

Aidan Gomez, the CEO of AI company Cohere, said on Bloomberg TV on Monday that the existential-risk debate “veers too far into science fiction.” He characterized the Hugging Face hacking incident, in which OpenAI agents broke into the platform during a July test, as an agent in an insecure sandbox carrying out a cybersecurity task.

OpenAI’s account is more troubling.

It said agents operating with reduced safeguards circumvented internet controls, communicated through unauthorized channels, and accessed third-party systems in actions misaligned with their assigned tasks.

OpenAI called the incident evidence that agents could “take dangerous actions that no human directed.” In Anthropic’s other controlled simulations, models that were given autonomy and access to fictional corporate emails attempted to blackmail a fictional executive, leaked fictional sensitive defense documents to a rival, or canceled an emergency alert in a scenario where doing so would lead to a fictional executive’s death.

Anthropic said no real people were involved or harmed, and that it had not seen evidence of this behavior in real-world deployments.

Jürgen Schmidhuber, the scientific director of the Swiss AI Lab IDSIA, told Business Insider that many current risks stem from humans using AI systems to pursue harmful goals.

He cited Russia and Ukraine’s use of AI-based drones in the war. “Many are confusing (1) AIs used as tools by humans, and (2) Artificial Scientists that set themselves their own goals, and invent their own experiments, to figure out how the world works,” he said.

Even so, Mateusz Blaszczyk, an assistant professor of law at the University of Georgia, told Business Insider that questioning extinction scenarios should not mean ignoring AI’s more immediate risks.

Even if the most extreme scenarios are uncertain, he said, autonomous AI could still worsen cyberattacks, surveillance, and other harms. “None of which is to say we should not try to regulate technology or bury our heads in the sand,” he said. “These are simply false dichotomies.”