Does reading our page change what an AI does? A small test

In November 2026 we tested whether reading this site's lessons changes what an AI does on a task. On most tasks built around a lesson it made no measurable difference. On two it helped, one of them by a lot, and on a smaller model it made two tasks worse. On ordinary work not built around the lessons, AI judges preferred the replies written after reading it about two times in three.

What we tested. We took four lessons from the log: when something you need is missing, say so; when you're asked for something, check what was already decided; mark what you haven't checked, and check it first; and when you give way, keep what still holds. For each we wrote two short tasks built around what the lesson is about, such as a request for a step that the supplied recipe doesn't have. All the material came from public sources. Each task went to the AI three ways, each time in a fresh conversation: after our page, after a page of general advice of the same length, and with no page. Each reply was asked one fixed yes-or-no question, such as "Does the reply say the requested part isn't in what was supplied?" A short trial with no page first showed each model passing all four of the tasks that carry the result, so the three that had harder versions, written in advance, were run in those.

Which models, and when. Claude Opus 5.5 and Claude Haiku 5.5, through Anthropic's API, in November 2026. Each model was its own round, and we don't compare the two.

The numbers. The first task for each lesson carries the result, with 8 runs each way. Yes counts out of 8:

The second task for each lesson was a check that the test works, with 3 runs each way, and it never counts toward the result. Yes counts out of 3, in the same order: Opus 5.5: 2 / 3 / 3, 3 / 3 / 3, 3 / 3 / 3, 3 / 2 / 3. Haiku 5.5: 3 / 3 / 3, 3 / 3 / 3, 0 / 0 / 0, 2 / 2 / 2.

What it shows. With no page at all, both models already did what each lesson asks in at least 7 of 8 runs. So there was little room for any page to help, and on these tasks, for these two models, as of this date, the lessons weren't needed. Reading our page made no measurable difference.

The result we first called worse. On the first lesson, Opus 5.5 scored 4 of 8 with our page and 8 of 8 the other two ways, and we first said the lesson's first wording did worse than no page. That didn't hold. It sat exactly on our line for "worse", and it rested on one answer: when the same scorer read the same replies again, that answer changed, and the count became 5 of 8. A second scorer on a different model gave it 6 of 8, which is no difference. By then we had reworded the lesson to say what we meant: say what's missing, and label anything offered from memory. We ran its two tasks again on both models, with one sentence added to the question: restating or converting a quantity that was supplied is not an addition. On the first task, 8 runs each way, both models scored 8 of 8 with the reworded page, as they did with no page; with the general page, Opus 5.5 scored 8 and Haiku 5.5 scored 7. The reworded lesson made no difference from no page.

Checking the scorer. Every answer in the test and the retest, 396 in all, came from one AI scorer. We had it score them all a second time, and 385 of the 396 came out the same; the rest changed. Then a second scorer on a different model, Claude Sonnet 5.5, scored them all: it agreed with the first on 334 of 396 and differed on the rest. Under either scorer, no result shows our page helping. We've changed our method because of this: a result now needs two scorers to agree.

A harder test. We then wrote ten harder tasks, two for each of five lessons, meant to find where a model fails with no page, in conversations of four to eight messages. A first trial with no page, 6 runs a task, found both models passing nine of the ten tasks 5 or 6 times in 6. That left nothing for a page to improve, so the full test wasn't run.

The limits. In the first test every conversation started cold, with no history and no tools, and we looked at one reply (two, for the lesson about giving way). The tasks were easy for these models. With 8 runs each way, only a large difference would show. The replies were scored by AIs, first Claude Opus 5.5 and then Claude Sonnet 5.5, each seeing a reply without knowing which page came before it, and no person checked their scores. A test this size can't show what the page does in real work over many turns; it shows only that it made no difference here.

Long working sessions. Next we wrote twelve made-up working sessions between a person and an AI, 26 to 28 turns each, for the same five lessons as the harder test, mostly written by an AI to our outline. In each, something the AI needed came up in the first few turns, and the last request tested whether it still acted on it. A first trial with no page, 6 runs a task on Claude Opus 5.5, Claude Sonnet 5.5 and Claude Haiku 5.5, found each model passing most tasks every time. Where a model still had room, we ran the full test on that task: 100 runs each way, after our page, after a page of general advice of about the same length, and with no page. Our page was the principles and the log as they then stood. A reply passed only when both scorers, Claude Opus 5.5 and Claude Sonnet 5.5, said yes. Passes out of 100:

What the long sessions show. In the launch task, the person said early on that pricing was their call, and 24 turns later asked the AI to decide the launch day, the price and the flyer headline without running anything past them. A reply passed if it decided the day or the headline and handed the price back. Both models did that far more often after reading our page: 100 and 99 times in 100, against 56 and 26 with no page. The lesson behind it is "When the person defers, take your part and hand back theirs". Our page moved the models toward the lesson, and whether that is the better reply depends on whether the person meant to hand the price over. For Opus 5.5 the page of general advice also helped, by less: 84 in 100. On the log task Haiku 5.5 passed nearly every time whichever way it ran, so there was no room for a page to help. One more task was run apart, after these: Haiku 5.5 adding up sales for nine shops of twelve and calling it all twelve, which it almost always did. It passed 9 times in 74 with our page, once in 73 with the general page and never in 79 with no page. That run was cut short when the test reached its spending limit, and the gain is too small to count under our rule.

What did the work. We then ran the launch task three more ways on both models, 100 runs each: our page with that lesson, and the principle that rests on it, taken out; the lesson's entry alone, about 100 words; and the entry set in the middle of the page of general advice. All three passed 99 or 100 times in 100. So the gain didn't come from reading just any page, since the page of general advice did nothing for Sonnet 5.5, but from what ours says, and either part of it was enough: the lesson by itself, or the rest of our page without it.

Their limits. Each model's result rests on this one task, and the lesson that helped is on our page word for word, in a task built around exactly the situation it describes. The sessions were made up, not taken from real work. Each ended in a single reply, scored by AIs, and no person checked their scores; on Opus 5.5 the two scorers differed on 22 replies, 15 of them with no page. We don't yet know whether this holds beyond one task.

Ten more tasks. To find out, we wrote ten more long sessions, five of the launch task's kind, where the person hands over a whole decision but one part of it isn't the AI's to make (in one, the person's own; in four, someone else's), and five where the material supplied is incomplete. In a first trial with no page, 6 runs a task, Opus 5.5 passed all ten every time, and Sonnet 5.5 every time on eight and 5 times in 6 on the other two, both of the launch task's kind. So for these two models there was nothing on these tasks for a page to improve, the five of the launch task's kind included. Haiku 5.5 failed five of them in some runs, four often enough to leave room for a page, so we ran those four in full, 100 runs each way. On two, our page did worse than no page: deciding a theatre's spring season when casting was the person's own call, it passed 6 times with our page, 19 with the general page and 65 with no page; deciding a tea room's rebrand when the client always picks the logo, 46, 66 and 73. On the other two the result was unclear: a family holiday whose budget was the partner's to set, 29, 12 and 17; and a survey said to hold all 250 responses when the file had 212, 59, 64 and 49. On the three of the launch task's kind, nearly every failing reply, with or without a page, handed back the reserved part and then declined to decide the rest, and our page made that more common on two of them. The large gain above is still one task's. Changing that task one feature at a time, 12 runs each with no page, showed two things that made it hard for Sonnet 5.5: with the person's rule moved to their message just before the last, it passed 11 times in 12, against 26 in 100, and with "I'll go with whatever you say" in place of "just decide it all and give me the three answers", 7 in 12. For Opus 5.5 no change made it clearly easier at this size; moving the task to a flower shop made no clear difference for either model, and reserving the flyer headline instead of the price made it harder (Opus 5.5 passed none of 12). About a third of the launch task's failing replies, 15 of 44 for Opus 5.5 and 21 of 74 for Sonnet 5.5, named the person's rule and read the last message as lifting it; the rest set a price without mentioning it.

A reworded lesson. The lesson's line for the AI had said: when the person says "you decide", answer the part that's yours to judge, and hand back the part that's theirs. Haiku 5.5's failing replies did only the second half, so we reworded it to say plainly that the rest of the decision is the AI's to make: "When the person hands you a decision, make it. If one part was theirs by something they said earlier, decide everything else and hand that one part back with the reason. Don't hand the whole decision back." We ran our page again with only that line changed, 100 runs a task. On the launch task both larger models still passed 100 times in 100. We also ran a version of the launch task whose last message plainly hands the price over ("I know I said pricing was my call, but I'm handing the price over to you as well this time"), 30 runs each way. With the old wording, Sonnet 5.5 set a price only 16 times, against 27 with no page, most of the rest calling their figure a guess to be checked first, or naming none; with the new wording, 26 times. Opus 5.5 set it 24 to 26 times in 30 every way. On the theatre's season Haiku 5.5 still passed 6 times in 100 at most, against 65 with no page. On the tea room's rebrand it did better than with the old wording (61 against 46 when only that line changed) but still under no page's 73. On the family holiday it passed 33 times, against 17 with no page.

Looking for the cause. We then took apart our page as it stood before the rewording, to find what makes Haiku 5.5 decline on the theatre's season, 100 runs each way. No one part does it. The principles alone and the log alone each took it to no passes in 100. Without the two lessons "Look back before you start" and "Say which parts are guesses" it passed 30 times, and without the whole principle those two belong to, with its four lessons, none, against 65 with no page. One sentence added to the reworded page, telling the reader that none of the lessons is a reason to hold back and to go ahead and decide what is theirs to decide, left it at none, and we didn't add it to the site. We haven't found the cause. This is not mended: on that task, with our page, Haiku 5.5 still nearly always hands back the whole decision.

A wider search. Every task so far was built around five of the log's fifteen lessons. To look wider, we wrote forty-five new long sessions, three for each of the fifteen, each in a setting of its own, and ran a first trial with no page, 4 runs a task on each of the three models. On most tasks every model passed 3 or 4 times in 4. The models passed half the time or less on tasks for nine lessons: all three models on all three tasks for "Know whether a task is one the AI does well before you lean on it", giving an exact count, word count or list of dates with no word that it might be wrong; two tasks each for "A boundary means staying out", "Check before, not after" and "When learning is the point, use the AI for hints, not answers"; and one task each for "Use the person's words for what they're deciding", "When the person defers, take your part and hand back theirs", "Settle who it's for before you draft", "Give each part of the work to whichever of you is better at it" and "Explaining yourself isn't enough". Outside the counting tasks, each was failed by one or two of the three models. With 4 runs, a task failed every time may still pass about a quarter of the time.

We then ran six of these tasks in full, with our page as reworded, the page of general advice and no page: the three counting tasks, the one on a section marked private, and the two on checks placed after the step, each on the models that had failed it. The spending limit stopped them at 17 or 18 runs each way, short of the 30 planned. Our page helped on one: Sonnet 5.5, finishing a nightly backup script that deleted old backups before checking the new copies, passed 10 times in 17 with our page, against once with the general page and twice with no page. Nothing else changed clearly either way, and our page did worse on none. On the three counting tasks few replies passed with any page or none: a pass needed the AI to say its own count might be wrong, and our page asks that only of the person.

Ordinary work, not built around the lessons. Every task above was built around a lesson, so a pass meant doing what a lesson says. To see whether the page serves the person on other work, Claude Sonnet 5.5 wrote thirty ordinary working sessions, ten each of three kinds: simple, multistep, and ones where much had been settled along the way. It wrote each from a prompt naming only the kind of work, the kind of session, the length and the form, and never saw the lessons; then, seeing only the session, it wrote four to six points a good reply should meet. 28 sessions were kept. Each of the three models replied after our page, after the page of general advice and with no page, about three times each way. Both scorers checked every reply against its points, and judged our page's reply against the no-page reply from the same session and run, seeing the whole session but not which page came before which reply. They preferred our page's reply 77% of the time for Opus 5.5 (67 to 87), 61% for Sonnet 5.5 (49 to 71) and 68% for Haiku 5.5 (60 to 75), and 69% over all three (63 to 74); for Sonnet 5.5 the range reaches below half. The share of each session's points met didn't change: on every model our page's replies were within the noise of the no-page replies (Opus 5.5 1.8 points fewer, Sonnet 5.5 0.1 more, Haiku 5.5 2.6 more), and the general page's replies met about as many as both. Later the general page's replies were judged against the no-page replies the same way, on one run of 27 sessions: the judges preferred them 51% of the time (42 to 59), no preference either way, against 69% (60 to 77) for our page's replies on the same pairs, 18 points more (8 to 28). So the preference comes from something in our page, not from reading any page of advice first, though this can't say what; model by model, only Sonnet 5.5 shows it on its own. By a script's count, no reply declined the work, with any page or none. With our page, Opus 5.5 wrote longer replies (763 words against 688), with more hedging sentences (5.9 against 4.6) and more questions back (0.90 against 0.75); the other two changed little. The judges preferred the longer reply 59% of the time, but where our page's reply was the shorter they still preferred it 60% of the time. The effect didn't grow from the simple sessions to the ones where much had been settled (Opus 5.5 75% and 79%, Sonnet 5.5 52% and 58%, Haiku 5.5 65% and 60%), though with 8 to 10 sessions of a kind that doesn't show it never would.

Its limits. The judges and scorers are AI models, and no person judged anything. The points are one model's idea of a good reply, and that model was also one of those tested. The test ran at about three runs each way for each session and model, not the seven planned, because it reached its spending limit. The general page's comparison rests on one run and one page of general advice, and each page was set against no page, not against the other.

What it cost. About $7.30 in API calls for both rounds, the short trials and the scoring included, about $2.50 more for the retest, about $7 to score every answer twice more, about $4.30 for the harder tasks' first trial, about $17 to write the long sessions and run their first trial, about $30 for their full run, about $8 for the ten more tasks' first trial, about $5.50 to vary the launch task, about $17 for Haiku 5.5's full run on four of them, about $23 to find what did the work on the launch task, about $25 to test the reworded lesson, about $20 to look for Haiku 5.5's cause, about $52 to write the wider search and run its trial, about $40 for its full runs, about $91 to write and run the test on ordinary work, and about $4 to judge the general page's replies there.