Three Ways a Number Goes Wrong Before You Publish It
Authored by Andrew Maury
On a single day in July, three numbers in my company’s outbound material turned out to be false. None was a typo. None had been caught by review. All three had been read, approved, and sent.
I run a data studio that publishes execution-quality measurements on decentralized exchanges. Much of that work means putting numbers about other companies in front of those companies. A wrong number there does not cost a deal. It costs the only thing that makes the measurements worth anything, which is being right when it is inconvenient.
So the three failures were worth taking apart. They turned out to be one mistake wearing three costumes.
-
Counting rows instead of things
We had claimed “2,600+ proprietary labels.” The number came straight from the database and the query was correct. But 1,627 of those rows read “Manual Review Required.” They were placeholders for work nobody had done yet. We had counted the size of a table and called it a count of knowledge.
-
Reporting your own frame as a fact about the world
We said a major router showed zero volume on one network. Our system did say that. What it had actually measured was $2,525 against a true figure of $143,750, because our attribution missed the path the volume took. The sentence described our instrument and presented it as a description of the market.
-
Inferring something negative from an absence
We described a set of contracts as “non-canonical clones” because they were missing from a third-party label set. They were the exchange’s own official routers. Absence from someone else’s list is evidence about that list.
Why review does not catch these
Read those three claims again as sentences. They are clear, specific, and confident. There is nothing to trip over. That is the problem. This class of error survives review because reviewing means reading, and reading only confirms that the sentence says what the author meant. It cannot tell you whether the thing underneath is true.
A second reader catches typos and tone. A second reader does not re-derive a number’s provenance, and that is the step that would have caught all three.
What worked instead
We stopped relying on care and wrote a gate: a script every claim about another company has to pass before it can be sent. It refuses the three moves above by name. It also enforces sample floors, so a rate computed on too few observations cannot be quoted at all, and a staleness rule, so any figure older than two days gets re-verified rather than reused.
The staleness rule is the part that has earned its keep. Re-deriving two drafts this week surfaced two errors that had survived three previous rounds of review. One sentence said our fills came in 1.04 basis points better than a benchmark. The measuring tool’s own legend defines that sign as worse, so the draft had inverted its own argument while reading perfectly well. The second called a figure “the lowest among venues we classify as routers,” which was true on the day it was written. On current data three venues were lower. That claim had not gone stale. It had gone false, which is worse, because a stale number drifts and a false one just sits there sounding authoritative.
Note what did the work there. Not a sharper reader. A rule that forced the number to be computed again.
The takeaway
If your work involves handing numbers to journalists, the useful question is not whether you are careful, because everyone is careful. Ask instead which of your claims would fail if someone re-derived it from source this morning, and whether anything in your process would tell you before they did.
Three checks that cost almost nothing:
For every count, ask what a single row represents. If some rows are placeholders, you have a table size rather than a count.
For every figure your own system produced, decide out loud whether you are describing the world or describing your instrument, and publish the second as the second.
Date every number you send out, and treat superlatives as expiring by default. “The highest,” “the only,” “the first.” Those do not decay gently. They flip.
A quote in a published article cannot be edited later. The number will outlive your memory of where it came from, so the check has to happen before it leaves, and it has to be something other than reading it one more time.
Author Bio: Andrew Maury is Founder & Senior Data Scientist at Rantum, a data science and ML studio in San Mateo, California. He builds ClearTrace, which measures what decentralized exchange venues quote against what their users actually get.