False intelligence reached military command channels

The United States narrowly avoided boarding a Chinese ship after an intelligence report generated with help from an AI chatbot incorrectly claimed that the vessel was carrying components linked to a nuclear weapons program, according to reporting by CNN cited by Ars Technica and TechCrunch.

The operation was being prepared during the war with Iran. Military aircraft were already in the air when US officials discovered that the intelligence behind the planned operation was false, TechCrunch reported. The United States was preparing to intercept and board the vessel with air support, but the operation was aborted after the error was identified.

One source cited in the reporting described the episode as something that “almost started a war.” The sources did not indicate that the ship was actually carrying nuclear weapons or components for a nuclear weapons program.

Chatbot misidentified the ship’s manifest

The false report originated with an analyst at US Special Operations Command who used a chatbot to synthesize information about the Chinese vessel. According to the reports, the analyst asked the system to combine open-source intelligence with classified signals intelligence held by the government.

The chatbot inaccurately identified the material listed in the ship’s cargo manifest. The analyst then used the tool again to format the incorrect findings into an official-looking intelligence summary. That document circulated through command channels before officials recognized that the chatbot had produced a false conclusion.

The episode illustrates how an AI error can become more difficult to detect when it is converted into a polished report and passed through established chains of command. The initial output was not merely an informal answer; it was used in an intelligence product that influenced preparations for a military operation.

Experts warn about decisions involving force

The incident comes as military organizations increase their use of generative AI to accelerate analysis and decision-making. The Pentagon has described AI as providing an advantage by speeding up its “kill chain,” allowing commanders to respond more quickly. However, the same speed can allow an incorrect answer to move through an organization before it receives sufficient human scrutiny.

“It’s important for service members to understand the uncertainty inherent to LLMs,” Jake Steckler, a research scholar at GovAI and a veteran US Army officer, told TechCrunch in a written response. He said that understanding the uncertainty is especially important for decisions involving the use of force, including targeting, intelligence analysis and operational planning, because those decisions can have life-and-death consequences.

Steckler said the episode should lead to stronger safeguards rather than an outright rejection of AI tools. He warned that prioritizing rapid adoption above other considerations could create incidents that damage service members’ trust in these systems and ultimately slow their use.

US military AI adoption is accelerating

The reported near miss follows a broader US effort to deploy generative AI across military and intelligence work. In January, the Department of Defense announced an AI acceleration strategy intended to make appropriate data available across federated information-technology systems, including mission systems across the military services and other components.

Defense Secretary Pete Hegseth said while introducing the initiative that AI is only as good as the data it receives. The department announced last December that Google’s Gemini for Government would form the basis of its bespoke GenAI.mil platform. In August, it added Grok for Government as another option. Anthropic also provides a customized version of Claude for US spy work.

A Pentagon representative told Congress in June that the department uses generative AI to help create reports required by Congress. The department has also said that 1.5 million active personnel have used the military’s generative AI tools.

Human oversight remains a stated requirement

A 2023 State Department declaration on responsible military use of artificial intelligence and autonomy said armed forces should carefully consider the risks and benefits of AI while minimizing unintended bias and accidents. It also said accountable use should include a human in the loop and a responsible human chain of command and control.

The reported Chinese-ship incident raises questions about how those principles are applied when AI-generated analysis is incorporated into operational intelligence. Large language models can produce confident but unsupported information when they lack sufficient context, a behavior commonly known as hallucination. The reports also come amid wider warnings from AI researchers about severe risks from advanced systems and calls for stronger safety research and regulation.

The operation was stopped before the planned boarding took place, but the episode shows how an incorrect AI interpretation of intelligence can quickly become a military crisis when it is treated as reliable evidence.

Sources

• Ars TechnicaAI hallucination of Chinese nuclear components almost led to US military attack
• TechCrunchAI Hallucination Nearly Triggers US Military Operation