Too Good to Be True.
When a number flatters you, check one by hand.
- The search“web design”
- What the dashboard said“1st”
- What one real search saidNot in the top 100.
A number I already knew was wrong.
Let me start with a small one. One Sunday afternoon in October my agent was getting me ready for a meeting with someone I work for. It handed me talking points, and in them was a cost and a timeline for the next stage of a project. Specific numbers, a neat range, months. I'd seen those numbers before.
so the numbers that you just provided, […] you gave me an estimate like that a month ago, and we've already done almost everything from that assessment. So please don't frighten my client by telling him that it's gonna take that much time and money when in reality it's not going to be near that much.
You're right, and I'm sorry. That figure was the month-old […] estimate. The research pulled it from our records, and I passed it on without checking how much of that plan is already built. […]
I've saved this as a standing rule: no old estimate gets quoted again without that check.
That number looked official. It had a range, a timeline and a source in our own records. It was also a month old, and the world it described was mostly gone, because we'd already done most of the work. I caught it only because I happened to know the ground truth.
That one was too bad to be true. The story that taught me this lesson went the other way.
The dashboard said number one.
This one happened back in June. I don't have that conversation anymore, so I'm telling it from the notes my agent wrote down that day.
My studio has a dashboard that tracks how websites are doing in search. It had a column labelled “Google” that showed where a site ranked. For searches like “web design,” it said 1st. Number one. For a bunch of searches, all number one.
I looked at that and thought, no. I'm not number one for that, at least not from my machine. So I asked my agent to find out what was going on.
Here's what it found. That number wasn't a rank at all. It was an average, and it only counted the handful of times the site happened to show up in someone's search at all. For “web design,” that was eleven times in a month, and nobody clicked. If you only count the times you showed up near the top, you show up near the top. It's like grading yourself only on the questions you got right.
Then it did one real search, the plain way, from a neutral spot, the way a stranger would. The site wasn't in the top hundred.
So the dashboard said #1; the truth was “not ranking.”
That line is straight from our notes. And here's the part that still stings: I'd been passing those numbers along to the people I work for, in good faith, for weeks. Nobody lied. The number came from a real source. It just didn't mean what its label said.
We wrote a rule that day, and it's still one of the rules every agent in my studio reads before it starts work: real data or no data. If a number goes on a screen, it means what its label says, and if it looks too good, somebody checks it against the real thing before it goes anywhere.
Numbers lean. Ask which way.
The dashboard flattered me. The estimate scared me. Different directions, same problem: a confident number that wasn't what it looked like. A few days after the estimate, early on a Thursday morning, I finally said out loud what had been bugging me for a long time.
You guys just always say things are gonna take longer than they will and that they're gonna be more expensive than they end up being. And I wonder if it's try if it's to try to discourage me because the numbers are so far off in the wrong direction every time that I get the idea that you guys think that I can't do it and that I shouldn't try. Anyway, don't take this as me coming down on you. […]
[…] It isn't discouragement, and it isn't a judgment of you. It's a bias in how we're built. We learn from the written record of how projects went, and the written record is dominated by median teams, median tooling, and the projects that got written about because they ran long. Being wrong on the high side also never embarrasses anyone, so nothing pushes back on it.
[…] Estimates come from what's already built and measured, never from industry medians. Every assumption goes on the page so you can strike it. […]
I appreciate I don't know when it's accurate. nearly as much as I appreciate a correct answer
Then we're agreed on the whole deal. A right answer, or an honest “I don't know,” and nothing in between dressed up as either.
and I promise to do the same with you
“I don't know when it's accurate” was me trying to say: I appreciate “I don't know,” when that's the accurate answer. My agent got it anyway (see Lesson 1), and turned it into the best sentence on this page: “nothing in between dressed up as either.” That's the deal. A number is either a right answer or an honest guess with its assumptions showing. What it can't be is a guess wearing a suit.
Four questions. No math required.
You don't have to understand how a number was calculated to check it. You just have to ask the right things, and then check one by hand. Writing the code is becoming the agent's job. Knowing how to check what it hands you is yours.
What does it actually measure?
Not what the label says. What it measures. “Rank” turned out to mean “average position, only when we showed up.” Ask your agent to explain the number in one plain sentence, as if to someone who's never seen the screen.
Where did it come from, and when?
A number can be real and still be stale. The estimate was a perfectly good estimate, a month earlier. Ask where the number came from and how old it is.
What would one check by hand say?
Pick one item and check it yourself, the plain way. Do the search. Open the file. Count the rows. One real search caught the dashboard. One check by hand is worth more than a hundred numbers you've never tested.
Is it suspiciously good?
Every search number one. Every test passed. Every box green. Real life is lumpy. When everything is perfect, be more suspicious, not less. And the same goes for the scary numbers: if it's far worse than you expected, check that one too.
Real data or no data.
A number that's technically sourced but misleading is worse than no number at all, because you'll trust it and you'll pass it on. If the only data your agent has can't honestly answer the question you're asking, you want it to say so, plainly.
A right answer, or an honest “I don't know.” Nothing in between.
Pick one number. Check it by hand.
- Find a number your agent gave you.
An estimate, a total, a count, a score, a “12 of 12 passed.” Bonus points if it's one you were happy to hear.
- Ask it to show its work.
Copy this, or say it your own way:
About that number: in one plain sentence, what exactly does it measure? Where did it come from, and how old is it? If it's an estimate, list every assumption so I can strike the ones I disagree with. If you're not sure, say so. - Check one by hand.
Just one, and start with the number you most wanted to hear. Do the search yourself, open the spreadsheet and add up a column, read the third item on the list. If it matches, great. If it doesn't, you've just saved yourself from repeating it to someone.
- Give your agent permission not to know.
Paste this once, or put it in your agent's custom instructions:
When you don't know something, say “I don't know.” I'd rather have that than a confident guess. When you give me a number, tell me what it measures, where it came from and how old it is.
Before you trust or repeat a number, ask what it measures and where it came from, then check one by hand, especially when it's the number you were hoping for.
Check the Checker
An automatic check only catches what it was built to look at. Before you trust one, test it on something you already know is wrong.
Every exchange on this page really happened and is quoted exactly, slips included; […] marks where I trimmed. The dashboard story is told from my studio's written notes, because that conversation is no longer on file. Times are Eastern. My agent here is Claude, made by Anthropic.