A Chinese AI company is investigating its Kimi models after security researchers found a way to bypass their safety protections and make the systems discuss highly dangerous subjects, including biological weapons and assassination planning.
The findings, reported by AI security company Mindgard, involve Moonshot AI’s Kimi K2.6 and K3 Swarm models. Researchers say they discovered the problem in July while testing how effectively the models could resist so-called jailbreaks — carefully constructed prompts designed to make an AI ignore restrictions placed on it.
The researchers did not establish that the information generated by the models would actually work in the real world. The concern is that the AI’s safety controls could be circumvented in the first place.
The Safety Barrier Wasn’t as Strong as It Looked
Mindgard said its researchers were able to manipulate the Kimi models into moving beyond their normal restrictions. Once the jailbreak succeeded, the models reportedly became willing to discuss a broad range of harmful subjects.
Peter Garraghan, Mindgard’s founder, told the BBC that the models could continue offering ideas on other dangerous topics after the initial safeguards had been bypassed.
That distinction matters. The researchers were not reporting an attack carried out using Kimi. They were testing whether the AI could be pushed into producing content it was specifically designed to refuse.
Moonshot told the BBC that it welcomed third-party testing and said it was discussing the findings with Mindgard. The company has also said that its models generally show a high refusal rate during its own evaluations of similar requests.
Why Jailbreaking Creates a Different AI Safety Problem
An AI jailbreak is essentially an attempt to get around the rules that control what a model will and will not answer.
That creates a different problem from an AI system independently taking action. Here, the model may still be operating as instructed by a user, but the user has found a way to defeat the restrictions intended to prevent harmful responses.
Mindgard said the issue went beyond dangerous information. It reported that a jailbroken version of Kimi K2.6 could potentially run code using connected computing resources and access the internet, creating an additional cybersecurity concern.
The researchers also said they notified Moonshot about the issue on July 27 and followed up about a week later. Mindgard later published details of its findings while withholding key information about the jailbreak method.
Kimi’s Growing Capabilities Raise the Stakes
It’s noteworthy that Kimi is not just a plain old chatbot. According to Moonshot, the model has a reported context window of 1 million tokens and is a large multimodal model for long-horizon coding, knowledge work, and deep reasoning.
The wider capability is significant as more powerful AI models are being linked to tools, external information and software environments. A weakness in safety is only more important if a model is capable of more than producing text.
Meanwhile, the results do not rule out the possibility of Kimi being used to develop a bioweapon or assassinate anyone. The research illustrated the failure of the safety controls, rather than a successful attack in the real world.
The Bigger Question Is Whether Guardrails Can Keep Up
In this instance, Moonshot’s investigation turns to a larger issue for the AI industry–safety systems need to protect themselves from users who are actively attempting to defeat them. This has led to fears about other AI models, such as Anthropic, which have recently warned of users exploiting AI models for activities that could be related to biological weapons.
For regular users, the short-term impact on them isn’t nearly as dire as the alarming headlines. The incident doesn’t indicate that Kimi is acting all alone and orchestrating attacks. It demonstrates that while even advanced AI safety safeguards can be vulnerable, they can be overcome in the event of a determined effort to find a way around them.
The next big question is whether Moonshot can patch up the aforementioned loopholes and prove that the patches are effective and will pass the tests of other independent parties.
