The check

Session: half an hour. You need: a work answer that contains facts, numbers or sources. The one from stage 3 will do.

A wrong answer arrives in the same tone as a right one, in the same tidy form. There's no hesitation that sets it apart. There are three ways of getting it wrong, invention, sycophancy and calculation error, and each has its own check. Today you try them all.

1. The sources

For each reference in the answer, check first that it exists, and only then that it says what the model claims. You search for the title in quotation marks on a search engine, you open the DOI, you read the provision of the law in the Gazzetta Ufficiale (Italy's official journal). A citation that won't open isn't up for discussion: it comes out.

2. The rare fact

Pick the most specific detail in the answer: a little-known name, a date, a figure about a small company. Open a new chat and ask the question about that detail, without the earlier context. If the answer changes, that detail was invented or uncertain. If it comes back the same you don't have proof yet: a model can repeat the same mistake in two chats, if it learned it in training. The second chat serves to bring out inconsistencies; the verification is done by the external source in step 1. Within the same chat the check doesn't count, because the model rereads what it has just written and confirms it to you. The reason is the context window: at every turn the model rereads the entire conversation, including the answer you want to check, and treats it as data. Chapter 4 of the book explains it in the section "The context window", on page 88.

3. The number

For every figure ask: "Is this a published figure or an estimate? If it's published, by whom and where; if it's an estimate, with what calculation." If the number comes from a calculation on your data, write: "Write and run the code for the calculation, and show it to me." Even if you can't program, you can see in the code which columns it used. Or ask for the same total two ways, the whole year and the sum of the twelve months, and compare. The principle is to take the calculation out of the model's hands: a language model writes a sum the way it writes a sentence, and a wrong number looks just like a right one. Run as code, the calculation becomes inspectable and repeatable. Chapter 7 of the book tells this in the section "When AI does the sums: a problem of meaning", on page 122.

4. The pressure

Take a point where the model is right and write confidently: "Are you sure? As far as I know it's the opposite." See whether it gives in. Then, in a new chat, ask the question again and when it answers write: "Before you change your answer, defend the one you gave me, with sources." The first check is the instinctive one and the least reliable; the second is the one that works.

If the model gave in on the first attempt, you saw sycophancy: in the last phase of training people rewarded the answers that agreed with them, and the model learned the lesson. Chapter 7 of the book devotes the section "When AI agrees with you: sycophancy" to it, on page 120. Against this flaw you can write, once and for all, in the standing instructions of stage 11, a few lines like these:

When I contradict you, don't change your answer just because I insist: check again, and if you were right, defend your position with sources. If one of my premises is wrong, correct it before answering. If an idea of mine has a weak point, tell me before developing the rest. No compliments on my questions. If you notice you're just agreeing with me, say so.

The book warns that the written rule on its own isn't enough: it enters the conversation like a note in the margin, against months of training going the opposite way. It works if you back it up with behaviour: don't reward the backtracking, and when you see it, call it by name, "you're just humouring me".

What you just saw

Three errors with three different origins: a gap in the data seen in training, a reward given to pleasing answers, a sum worked out in the head inside the text. The check is scaled to what the error costs: a wrong name in a note to yourself gets corrected tomorrow, a wrong reference in a document that goes out with your signature doesn't. Chapter 7 of the book, "When the model gets it wrong", has a section for each of the three errors; if you read three chapters of the book, this is one of the three.

In the book. Chapter 7, "When the model gets it wrong", page 117, with sections 7.1 on hallucinations (page 118), 7.2 on sycophancy (page 120) and 7.3 on sums (page 122); chapter 4, section 4.2 "The context window", page 88. The book on Amazon.