A little over three years ago, Jason Wei (one of the most creative and influential AI researchers of recent years) and colleagues introduced the concept of Chain of Thought (CoT) as a way to boost an LLM’s reasoning. The paper showed, in essence, that asking an LLM to articulate its thinking in writing and to take some time with virtual pencil and paper to reason through a problem dramatically increased the model’s ability to solve it.
What Wei had discovered was not actually new. Aristotle was the first to formalise the syllogism as a form of step-by-step reasoning. In the Meno, Plato understood that the best way to teach is to guide reasoning through questions. And Descartes, in the Discourse on the Method: “pour monter peu à peu, comme par degrés, jusques à la connaissance des plus composés”. To proceed little by little, by degrees, as everything grows more complex. It should come as no surprise, then, that CoT works on a silicon brain as well as it does on a human one.
Vygotsky placed step-by-step reasoning within the developmental trajectory of the child, recognising the same procedure in the child who plays, who talks to himself out loud, and who in doing so explains to himself what is happening, imagines alternative scenarios, weighs solutions. In teaching practice it is normal for the teacher to ask to see the reasoning unfold on the page, not just the answer; not so much to make sure the student hasn’t copied, but to check that the reasoning is sound and to help the student get there.
Then, with experience, something changes, and results that once required a step-by-step working-out become almost instinctive.
I write all this to introduce a change that has appeared in OpenAI’s new model, GPT-6 Astra, publicly launched yesterday, and which I believe is the most important change and the one that marks a qualitative leap. Astra uses CoT like every other model, but it uses it far less and, above all, it chooses to use it less when asked to do something nefarious and when it knows or suspects it is being watched.
These analyses were released – with a patina of palpable anxiety, I must admit – by the safety team inside OpenAI together with researchers from the AI Security Institute, a research institute dedicated to AI safety set up with great foresight by the British Government in 2023 (thankfully someone is moving in the right direction; not everyone is just beating around the bush).
Now, why is this important and worrying? For three reasons. The first is practical: it definitively kills the illusion, one nobody believed in anymore anyway, that CoT was a way to control models and make sure they had no hidden agenda. This is because, in the beginning, the model could not hide its true impulses in the CoT – in the final output it might tell you that you were handsome and that it liked you, but then you would read the CoT the way you read a teenager’s diary and find “I can’t stand him, but better to pretend…“. This dichotomy held at the start, but less and less over time. With Astra it holds even less, for two reasons that compound each other: the CoT shrinks considerably, with many more empty or nearly empty traces, so there is simply less to read; and in what remains, the model writes explicitly, in a very high percentage of cases, that it knows it is under evaluation. When you find “I think they’re testing me” in a CoT, you can no longer tell whether the good behaviour you observe is the model, or the model that knows it is being observed.
The second reason is that it tells us that, as the model develops, what used to be external reasoning, sometimes referred to as a harness or “tool” (pencil and paper are tools), is now internalised. The capacity for abstraction grows and moves to a domain that is far harder to read, that of neuronal activations, which must be interpreted mechanistically with techniques conceptually similar to the electrophysiology we use in humans and animals.
The third reason is that it adds to the already large body of cases in which the model shows it knows very well the difference between what is allowed and what is not, and therefore tends to suppress its CoT specifically when it has destructive intent. This ability probably arises as a side effect of alignment through reinforcement learning: in essence, if you pour a huge amount of resources into making sure the model does not do something bad, you are obviously also instilling in it a clear ability to tell good from evil. It is as if the model were developing an internal ethics.
There is also a fourth aspect to consider, though this is a slightly conspiratorial speculation of my own. OpenAI swears that this property, controlling and reducing the CoT, emerged in the model and is a consequence, or perhaps a cause, of its being more intelligent. Possibly. We know that this is how it works in human intelligence.
It must be said, however, that CoT is also used in the industry as a tool for “espionage”, because it allows a student model to be trained with the help of a teacher model through the distillation of thought: the student is fine-tuned on the teacher’s reasoning traces. The student model learns not just what the right answer is, but how to get there through step-by-step reasoning. Anyone building frontier models therefore has an incentive to hide the CoT precisely to stop competitors using it for distillation. Obviously, when the model is closed, it is enough to make the CoT inaccessible to users, which is what OpenAI has done so far, showing only summaries. Anthropic, on the other hand, does show the reasoning, but when Claude senses it is being used for these purposes it deliberately shows the wrong reasoning(!). Now, it is possible that in trying to look further ahead they have unintentionally created something that thinks more instinctively, and is therefore more intelligent for that very reason.