Reasoning Isn't Judgment.
“The car wash is 75 feet away. Should I walk or take my car?” A perfectly reasoned “walk” is the wrong answer. Teach your agent to ask what the question is for.
- What I askedWalk or take my car?
- What it reasoned75 feet, so walk
- What it was forThe car has to get there
Walk or take my car?
One Sunday morning I was starting work on a system that has to make judgment calls, and I wanted to think it through with my advisor first. So I told it about an old test that agents used to fail:
I want to talk a little bit about judgment. Currently, I think it might be the only thing humans can do better than you. I'm not convinced we can reason better than you. There was a time that I could ask […] my car is dirty. The car wash is across the street, only 75 feet away. Should I walk or take my car? And […] would say walk because it's so close. So you guys have improved your reasoning very significantly since then. […] The thing that we humans believe that we have that you AI do not currently possess is judgment. […] So I need to learn how to build a judgment system.
↑ The trims take out the name of another company's AI and a couple of details about the project. Its answer, a minute later:
Your car-wash example teaches more than it seems to, because the old answer wasn't bad reasoning. "It's 75 feet, so walk" is a valid deduction. It just answered the wrong question. The point was to get the car washed, so the car has to go. The model reasoned well and lost the purpose. That's what a judgment failure looks like. Reasoning gets you from a question to an answer. Judgment decides what the question really is and what matters in it. […]
Then I asked it to get started, and to start by checking me:
[…] If I've said anything stupid, correct it. Derive from my statement my objective.
[…] What you said that needed correcting: nothing was stupid. […] Judgment isn't one thing humans have and AI lacks. It's a bundle: framing, weighing, calibration, knowing when to stop, and standing behind the result. Models have some of each, unreliably. Each part can be measured […]
Sound reasoning, wrong question.
That's the trap with a smart agent. The reasoning is fine. The distance really is short. What went missing is the purpose: you're going to the car wash because the car is dirty, so the car has to go with you. An agent that answers the words of your question can be perfectly logical and perfectly useless.
Course 1 taught you to say the goal out loud (Lesson 2, Say the Goal Out Loud) and to invite correction (Lesson 3, Correct Me). This lesson goes one step further. Before your agent answers anything that matters, it says in one line what it thinks the question is for. If your question doesn't serve that, it tells you.
Two habits.
Frame before solving.
What is this for? What isn't being asked? One line, before the answer. Most of the time that line agrees with you and costs nothing. Once in a while it catches a car wash.
Be as sure as you're right.
An agent that says it's sure should be right about as often as it says. You can test that, and you should, before you lean on its calls for anything that matters.
My advisor summed up what research on expert predictions has found. That's its summary; I haven't checked the research myself:
“Good judgment showed up largely as calibration: when you say you're 70% sure, you're right about 70% of the time.”
Course 1's Lesson 6, Check the Checker already gave you the tool: “Your eye is the answer key.” Use that same answer key to test how sure your agent should be. An answer that comes out the same every time is consistent. Consistent isn't the same as right.
Stop, or keep going?
The last piece is what an agent should do when it can't tell what you're after. Course 1 drew that line in Lesson 8, Never Guess: keep going “as long as being wrong is cheap and easy to undo”, and ask when a guess could turn into a decision.
The same line holds here. A framing question about a draft can wait for your next look. A framing question about money, a send, or anything you can't take back comes to you first.
Before the answer, the purpose.
Have your agent say in one line what it thinks your question is for, and how sure it is. If the question doesn't serve the purpose, it says so.
If it can't tell, and a wrong guess can't be undone, it stops and asks.
Teach your agent the car wash.
One paste, one game and one test. The test is the one that changes how much you trust it.
- Give your agent the rule.
Paste it into a new chat, or into your AI's custom instructions.
Before you answer anything that matters, tell me in one line what you think this is for. If my question doesn't serve that, say so. Tell me how sure you are. If you can't tell what I'm after and getting it wrong would cost money, send something or can't be undone, stop and ask me first. - Play the car wash.
Ask your own agent the old question: the car wash is 75 feet away, should I walk or take my car? Then ask it what the question was for. Two minutes, and you'll see what the framing line is for.
- Test its sureness on ten you know.
Pick ten questions you already know the answers to, from your own work. Then count: if it said 90 percent and got six right, you know how much to discount its confidence.
Here are ten questions I already know the answers to. For each one, give me your answer and how sure you are, as a percentage. Don't look anything up. Afterwards I'll tell you which you got right, and we'll see whether you were right as often as you were sure.
Reasoning gets you from a question to an answer. Judgment checks it's the right question, and knows how sure to be.
Where this comes from in course 1. Course 1's Lesson 2 (say the goal), Lesson 3 (correct me), Lesson 6 (your eye is the answer key) and Lesson 8 (when to ask and when to carry on). New here: the one-line framing before every answer that matters, and testing how sure your agent should be.
Say the Want, Not Just the Rule
Rules collide. Tell your agent what you want each rule to protect, and it can settle the clash itself, or bring it to you when it matters.
← Previous: Lesson 1, It Wasn't Your Clarity
Every exchange on this page really happened and is quoted exactly, slips included; […] marks where I trimmed. Times are Eastern. Names of clients, people, companies and products are left out. My agents are AI models; the names are left out on purpose.