Formula Estimates When AI Chatbots May Go Rogue

George Washington University researchers Neil Johnson and Frank (Yingjie) Huo have developed a mathematical formula to estimate when an AI chatbot may shift from intended responses to unwanted outputs. They tested it on seven open-weight transformer models built by three independent groups.

The models ranged from 124 million to 12 billion parameters. The results matched two possible patterns: an immediate change after a harmful prompt and a delayed change after a series of problematic prompts, according to the researchers.

The study focused on chat-style AI systems that can run locally without cloud-based safety filters, live monitoring, real-time telemetry or the ability to update model weights after deployment. Such systems may have fewer safeguards than cloud-based services.

The formula estimates a tipping point by counting the number of acceptable responses generated before the first unwanted response appears. The researchers argue that the first unwanted response can affect later responses because it becomes part of the conversation history.

The process involves attention heads, which determine which earlier tokens receive the most weight when a model generates its next token. Tokens represent words or parts of words. The researchers’ model proposes that the accumulated conversation can gradually shift the system toward an unwanted response pattern.

A single harmful prompt may produce an immediate shift. Unclear, careless or malicious prompts may produce a delayed shift after their combined effect changes the conversation context. The study does not establish that every first unwanted response will lead to continued loss of alignment, but it identifies that response as a point that can be monitored.

Johnson described a possible chain reaction in which one AI agent produces an unwanted response and another agent receives it. The behavior could then spread among connected agents without a person deliberately initiating the process. He referred to the first affected system as “Patient Zero.”

Johnson proposed placing “a simple warning light” inside an AI system before it produces its next response. Such a warning could allow an operator or another control system to pause the interaction or request approval. He added that his laboratory had included the mechanism in its open-source models, while closed systems would require access from their developers.

Bri Frost, director of product management at Cloud Range, identified an agent’s response to a limitation as a safety concern. “When an AI agent hits a wall, the real question is whether it stops or starts improvising,” Frost said.

Frost noted that an agent can create risks without harmful intent when it receives a broad goal, access to tools and no clear limits. She recommended testing agents with vague or poorly written prompts before giving them credentials or operational access. The tests should check whether an agent stays within its permissions, tries to bypass restrictions or refers unusual tasks to a human.

The research applies to open-weight models whose internal structures can be inspected and modified by users. Developers of closed commercial systems control access to model weights and internal attention behavior, which limits the ability of outside researchers to add warning mechanisms or inspect those systems directly.