AI hallucination nearly triggers US military operation

“It’s important for service members to understand the uncertainty inherent to LLMs,” said Jake Steckler, research scholar at GovAI and veteran U.S. Army officer in a written response to TechCrunch. “But it’s especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences for those decisions.” 

Still, Steckler says, the incident should serve as a call to add more safeguards to AI, not a reason to avoid it. “These tools can be useful in the right contexts and with the right safeguards in place,” he said. “But prioritizing adoption speed over all else will likely lead to incidents that only make service members lose trust in these systems, which ultimately is only going to slow adoption.”