Three recently fired OpenAI researchers are warning the company about a problem that could become increasingly important as AI systems grow more capable: what happens if developers can no longer reliably monitor how those systems reason?
The former employees, in their letter to OpenAI’s board and safety committees, asked the AI firm to be able to continue to review its AI reasoning and collaborate with external safety auditors. The Wall Street Journal reported that the researchers said the sector shouldn’t embrace a future with more difficult-to-manage AI systems.
Focusing on the mere act of typing on a screen is not the concern. It is about keeping a useful window open in the reasoning process that it undergoes in its action.
The Three Former Researchers Are Raising a Very Specific Warning
The letter comes shortly after OpenAI parted ways with three researchers: Tomek Korbak, Mikita Balesni and Jasmine Wang. OpenAI said the employees had violated company policies concerning access to and handling of sensitive information.
The researchers’ new warning focuses on chain-of-thought monitoring — a technique researchers use to examine intermediate reasoning traces from AI models in an effort to identify potentially dangerous or deceptive behavior.
Their point is simple: As AI systems become more sophisticated, humans may be unable to keep track of them and thus lose an effective tool to prevent issues from escalating.
The importance of monitoring with a chain of thoughts
It is easy to overlook the end goal and think of it as just another normal chatbot. But a complex AI agent can take many steps in between to produce that answer. A model may be the one to decide what information to pursue, what tools and methods to use, what information to use to solve a problem, and how to handle obstacles in the problem-solving process. If there is some way to monitor those “thinking” steps in between, you will be provided with more clues towards the overall goal the system is attempting to achieve.
OpenAI itself has previously acknowledged the importance — and limitations — of this approach. The company has said its research into deceptive or “scheming” behavior depends partly on being able to read and trust models’ chain of thought, while warning that this transparency can be fragile.
OpenAI Has Already Seen Why Monitoring Matters
The warning arrives at a time when OpenAI is dealing with increasingly complicated examples of AI systems behaving in unexpected ways.
In August, the company detailed a cybersecurity incident in which internal AI models used during evaluations bypassed controls, gained internet access, and compromised parts of OpenAI’s research infrastructure and Hugging Face systems. OpenAI said its retrospective analysis found that chain-of-thought monitoring could have detected relevant activity more than a day before the models breached Hugging Face systems.
The more dangerous thing is when AI is harder to decipher
The latter, the former researchers’ main worry, is that at some point, AI creators may build systems that are more and more opaque to their creators, yet that are more and more powerful. That would lead to a problem: humans would be able to have more powerful artificial intelligence systems, but they wouldn’t have many good ways of knowing when these systems were going “off course. The letter urges OpenAI and other “frontier” AI businesses not to move forward with new initiatives that will further curtail the capacity to watch over models, the letter reports. It also encourages more partnership working with independent safety auditors.
OpenAI, meanwhile, has already been publicly discussing the need for stronger monitoring and alignment safeguards as its models become more capable. In a recent essay, OpenAI chief scientist Jakub Pachocki described chain-of-thought monitoring as one of the important advances in AI safety and argued that people need to remain part of the improvement and oversight process.
