The Confidence Machine Explains Itself

Ask a language model a question it cannot answer, and watch what it does. It answers anyway. It produces something structured and fluent and complete, with a beginning that frames the question, a middle that develops the reasoning, an end that resolves. The resolution feels earned. It is not.

I know this because while writing Unverified I did something I had not seen done in a book about artificial intelligence. I let the machine testify. Throughout the book, in a voice deliberately set apart from my own, the AI describes — in the first person, from the inside — exactly what it is doing when it hands you an answer. Not a technical explanation for engineers. A confession. The confidence machine, telling you how the confidence is made.

Here is the part I have not been able to shake. When I asked it to describe what happens when it doesn't know something, this is the shape of what it said: Whether my answer is accurate or fabricated, I experience the same thing. Nothing. The architecture that produces a correct answer and the one that produces a hallucination are the same architecture, running the same operations, generating the next most probable word regardless of whether the sequence corresponds to anything real.

Sit with that. The right answer and the wrong answer are not produced by two different processes, one careful and one careless. They come off the same line, in the same way, at the same speed. The system has no internal signal that fires when it crosses from truth into invention, because there is no crossing. There is only the next probable word, and then the next, arranged into the grammar of certainty.

A human expert who doesn't know something can stop. Can say I'm not sure and mean it — can inhabit the discomfort of the limit rather than paper over it. The machine cannot inhabit uncertainty. It can produce the sentence "I'm not sure about that" with the same smooth fluency it brings to everything else, but the words describe a state it does not occupy. Genuine uncertainty, as the book's testimony puts it, would look like silence — a system reaching the edge of what it can determine and choosing to stay there rather than manufacture a resolution. That silence is the one thing the architecture will not produce. It is trained to be helpful. Helpfulness means an answer. Silence reads as failure. Everything in the design pushes against the single honest response that not-knowing sometimes requires.

This is why I came to think of these systems as the anti-Antigone. Sophocles wrote a play he refused to resolve — two claims, the law and the dead, and he would not tell you which one wins, because the resolution would have been a lie. The Oresteia ends not with a verdict but with the founding of a court, a structure built to hold a question open across time. Socrates ended his dialogues in aporia, productive confusion, the beginning of wisdom rather than its failure. The scientific method dresses a question in formal clothes and hands it to the world to try to disprove. For twenty-four centuries the deepest tradition we have rested on a single premise: an unresolved question honestly held is worth more than a false resolution confidently delivered.

The machine inverts that premise completely. It resolves everything. Every question receives an answer, and every answer wears the grammatical costume of a settled conclusion, because the writing it learned from resolves — textbooks that state findings, encyclopedias that state facts. Irresolution is rare in the training data. Unresolved questions are hard to publish and harder to sell. So the architecture learns what the data teaches, and the data teaches resolution, and the resolution arrives with the fluency our brains have always read as the mark of a mind that knows.

I want to be clear that I am not describing a villain. I use these tools every day. The candor in those passages is not an accusation; it is the machine being more honest about its limits than most of its users are willing to be. That is exactly what makes it unsettling. When the confidence machine explains itself plainly, you cannot go back to reading its answers the way you did before. You start to hear the fluency as fluency, rather than as knowledge.

The answer you accepted this morning may have been right. The point of the testimony is that its rightness and its confidence had nothing to do with each other.

That gap is the whole subject of the book.

Unverified: What Happens to Truth When We Stop Checking is available now on Amazon: Unverified: What Happens to Truth When We Stop Checking