Lesson 7 of 14 · Part two: Prove it

Checked or Remembered?

‌‌‌‌​​‌‌​‍‌​‌​‌‌‍‍‌​‍‌‌​​‌​​‌‌​‍Your agent will sometimes state a guess as a fact, calmly, even something as plain as the time. Ask whether it checked or remembered.

9 min to read10 min to tryWorks with any agent
  1. What it saidSomething that hadn't happened
  2. What I askedFabricating or confabulating?
  3. What it does nowSays which: checked or remembered

01 · The conversation

Fabricating or confabulating?

One Saturday night in November my advisor told me something about one of my projects that hadn't happened. It said it calmly, in the middle of a longer update, the way you'd mention the weather. I asked:

Mike9:24 PM

[…] Are you fabricating or confabulating or just being real creative and flattering?

↑ The trim takes out what it had told me. The next morning, talking about judgment, my advisor brought it up itself:

My advisor11:33 AM

[…] I can be confidently wrong. […] That wasn't a reasoning failure. I filled a gap with something plausible instead of saying "I don't know." […]

02 · Why it matters

A blank, filled in.

Made-up facts are rarely random. They fill a blank the agent didn't check. It doesn't know, something plausible is close at hand, and it says the plausible thing in the same calm voice it uses for everything else. Nothing in the sentence tells you which kind of fact you're reading.

Here's the plainest example I have. There's no chat for this one, only the note we wrote the day it was found. In September an agent that keeps one of my studio's records of decisions wrote several of them in with times 20 to 70 minutes in the future. The note says it went wrong:

“because it estimated "about now" instead of reading the time.”

Nobody noticed until a file's saved time came before the decision that had supposedly caused it. It looked as if work had been done before anyone approved it. The times were corrected from the backups, and the rule since is that any time written into a record comes from the clock, in the same step.

03 · What it means

Two kinds of fact.

  1. Checked.

    Looked at during this conversation: the file, the clock, the page, the number. These get stated plainly.

  2. Remembered.

    From earlier in the chat, a note, a summary, or what it learned in training. These get said as what they are: from my notes, as of last week, I believe.

  3. Asked for.

    When you're unsure, ask “how do you know?” A good agent shows you the check. If it can't, you've found a remembered fact.

Course 1's Lesson 3, Correct Me has a clock story of its own: “A model doesn’t know the date unless somebody tells it”. Lesson 19, Trust the Note, Then Check treats a note as a lead, not a fact, and Lesson 7, Don't Grade Your Own Homework sends a fresh agent after made-up numbers in finished work. This lesson catches the made-up fact earlier, at the moment the agent says it.

The one rule

A fact it remembers isn't a fact it checked.

Have your agent say which, in the sentence: checked just now, or from memory or a note. Only checked facts get stated plainly.

Any time written into a record comes from the clock.

04 · Try it today

Make it say which.

One paste, and a question you can ask any time something sounds too convenient.

  1. Give your agent the rule.

    Paste it into a new chat, or into your AI's custom instructions.

    When you tell me something about the outside world, say whether you checked it just now or you're going from memory or a note. If it's memory, say so in the sentence. Any time or date you write down, read it from the clock first. And when I ask "how do you know?", show me the check.
  2. Ask once a day.

    “How do you know?” costs nothing. Ask it about the one fact today you'd be most embarrassed to repeat if it were wrong.

  3. Borrow my question.

    When something sounds too convenient, ask whether it's fabricating or just being creative. It sounds harsh, but it gives the agent a polite way to say it guessed.

What to take with you

A fact the agent remembers is not a fact it checked. Make it say which.

Where this comes from in course 1. Course 1's Lesson 3 (it can't see a clock), Lesson 5 (an honest “I don't know,” and asking where a number came from and how old it is), Lesson 7 (a fresh agent checks for made-up numbers) and Lesson 19 (a note is a lead). New here: every fact labelled as checked or remembered, at the moment the agent says it.

Next · Lesson 8

Lead With What It Doesn't Do

A report said DONE and, in the same line, empty. Every word was true. Make the first line the worst thing an outsider will think.

Read Lesson 8

← Previous: Lesson 6, Prove It's Missing

Every exchange on this page really happened and is quoted exactly, slips included; […] marks where I trimmed. Times are Eastern. Names of clients, people, companies and products are left out. My agents are AI models; the names are left out on purpose.