The incident occurred when a Special Operations Command analyst tasked a chatbot with merging open-source data and classified signals intelligence. The system hallucinated the ship’s manifest, and the analyst subsequently used the tool to format the inaccurate data into a formal report. This erroneous summary circulated through command channels before reaching the mission-ready stage.
In section Startups & Technology
AI Hallucination Nearly Triggered Military Strike on Chinese Vessel
U.S. military aircraft were already airborne this spring when a mission against a Chinese ship was aborted, narrowly avoiding an international confrontation. The operation had been ordered based on intelligence synthesized by an AI chatbot, which falsely identified the vessel’s cargo as components for a nuclear weapons program.
This near-miss highlights the tension between the Pentagon’s push for high-speed decision-making and the risks posed by unverified machine output. While military leaders view AI as a critical tool for accelerating the "kill chain" against rivals like China, the episode exposes how quickly automated errors can bypass human oversight. Jake Steckler, a research scholar at GovAI and a U.S. Army veteran, warned that prioritizing adoption speed over rigorous verification could erode trust in these systems, potentially undermining the very military advantage the technology is intended to provide.
Comments (0)
No comments yet. Be the first!