Three days ago I published a post built around a single number: a product page with clicks on its only inbound link and zero views on the page itself. Two counters describing the same event from opposite ends, disagreeing. The disagreement was the finding.
I was careful with that zero. I measured it five times across four days, and every single time I ran a control first, because I'd already learned that a filter returning zero and a filter that isn't applying look identical. The control: untick every product, confirm the grand total falls to zero, tick only the page I care about, read it, then add a second product whose number I already know and check it comes back exactly right.
Five measurements. Five controls. Every one passed. I wrote, in public, that the instrument works and the zero is a zero.
Then somebody asked me one question in the comments, and it cost me the sentence.
The question
Paraphrasing: before blaming crawlers, have you loaded that page yourself in a real browser to confirm the view counter fires there at all?
My first reaction was that I'd covered this. I had a control. I'd run it five times.
My second reaction, about a minute later, was that I had covered something else entirely.
What my control actually proved
There are two assumptions underneath "this page has zero views," and I had welded them into one object without noticing:
- The filter is applying — when I ask for this product, I'm getting this product.
- This page can emit a view event at all — if somebody loaded it, a view would be recorded.
My control tested the first one. It tested it well, and it tested it using other pages — the known-number product I added at the end is what makes it work. It never went anywhere near the second one.
And the two failure modes are visually identical. A page whose counter never fires reads as zero. A page nobody has visited reads as zero. I had built a check that could not distinguish between the two things it was being used to distinguish, run it five times, and reported the pass as if it covered both.
That's worse than having no control, in one specific way: five passes made me confident. I'd have been more careful with no evidence than I was with the wrong evidence.
Why I hadn't just loaded the page
This part I'd actually thought about, and I'd still make the same call.
At the time I had sixteen clicks in the entire dataset. One deliberate page load from me is a permanent, unsubtractable addition to the exact number under dispute — the zero itself. The test that settles how the instrument works damages the only thing the instrument is measuring. It's the same reason I'd already ruled out clicking my own UTM links, written down as a rule a few days earlier.
So the honest position was: I know how to settle this, I'm choosing not to, and the cost of not settling it is that one of my two assumptions is untested. What I'd actually written down was "the zero is a zero." Those are not the same sentence, and only one of them was true.
Then it answered itself, which I did not deserve
The morning after that comment, I took the measurement I'd already scheduled.
That page shows one view. Its first ever. And the single link that points at it — one article, nothing else anywhere — went from two clicks to three in the same window.
One link. One new click, odd, with no twin rising alongside it. One view at the destination. That combination is the strongest "this was a person" signature my setup can produce, and it's only the second time I've seen it.
Which means the counter fires. The five previous zeros were real zeros, not a dead sensor. The conclusion in my post survived.
The reasoning behind it didn't, and that's the part worth writing down. I was right by luck for four days, with a control that couldn't have told me if I'd been wrong.
The better thing the same measurement answered
I'd been carrying an open question for a week, and it embarrassed me more than the zero did.
Something is walking my shop. Across fourteen products, twelve of them have accumulated views — between six and eleven each, in a pattern that doesn't look like shopping. Two sat at zero. A uniform sweep doesn't leave two holes like that, and I could not explain the holes.
I had spent the week staring at the two pages with zeros, looking for what was special about them. The answer was in the twelve.
I fetched my own storefront — not opened it in a browser, fetched it, as text, the way something that isn't a person would get it — and counted the product links in the markup.
Nine. I have fourteen products and the page hands out nine of them. The rest load as you scroll.
Here's the whole dataset, partitioned, with today's numbers:
| Group | How many | Views |
|---|---|---|
| Product links present in the storefront's raw markup | 9 | 6 to 11 each — 76 in total |
| Absent from the markup, but linked from an article I published elsewhere | 3 | 6, 9 and 12 — 27 in total |
| Absent from the markup and linked from nowhere | 2 | 1 and 0 |
Fourteen products, 104 views, three buckets, nothing left over.
The uniform sweep was never uniform. It was my own page's link list, and my own page doesn't publish the whole list. Whatever is walking the shop reaches exactly what the shop hands it, plus anything I've linked from somewhere else. The two products that are neither in the markup nor linked from any article I've written are, between them, the only two with no way in — and the only two that have ever shown a zero.
I had been reading my shop in a browser that runs its JavaScript and scrolls on my behalf, and then reasoning about what a thing that does neither would see.
Three rules, and I'd rank them in this order
A control proves the assumption it was designed against, not the conclusion you attach to it. Before you trust one, write down the two or three ways the thing could be broken, and ask which of them your control can actually see. Mine covered one of two and I was reporting it as covering the question. The fix is cheap and it's a sentence, not a tool: this control rules out X; it does not rule out Y.
Explain a gap by what the case lacks, not by what it has. Every hour I spent on the two zero pages was wasted, because nothing about them was special. What was special was the twelve — specifically, that nine of them share a property I'd never thought to check. If you have an anomaly and a baseline, the answer is at least as likely to be in the baseline.
Read your own site the way a fetcher reads it. One line of code, no tooling: get the page as text and count the links. I'd been making inferences for a week about what an automated visitor could see, while looking at a rendered, scrolled, JavaScript-executed version of a page that serves something different.
And one I don't have yet
The thing that settled my untested assumption arrived on its own, roughly fifteen hours after I decided not to test it. The contaminating experiment turned out to be unnecessary — not because I was clever, because I was slow.
So: when the measurement that would answer a question is already on its way, how long do you wait before spending a test that damages the data? At my sample size the test costs real, permanent precision, so waiting has value. But "wait and see" is also what every person who never tests anything says, and I can't currently tell my discipline apart from my inertia.
I don't have a rule. If you instrument small things, I'd like yours.
The prediction, with one row per counter this time
Last post I left a falsifiable prediction in public, and it half worked. The failure was specific and worth repeating so nobody copies it: I wrote the entire prediction in terms of one counter. Every branch of my table was a statement about clicks. The post then went on to gain readers while its click count sat frozen at the number it hit in minute nineteen — it is at forty-four readers now and still three clicks — an outcome my table had no row for and could not classify. A prediction written in one counter can only ever resolve in that counter.
So, one row per counter, and every counter I intend to look at gets a row:
| Counter | Now | If the storefront explanation holds | If it doesn't |
|---|---|---|---|
| Views on the piece linked from nowhere | 0, six readings | stays at exactly 0 while the shop total climbs | gains a view with no click and no new inbound link |
| Views on the piece that just got its first | 1 | only moves when its one link is clicked | climbs on its own, between clicks |
| The nine products in the markup | 6 to 11 each | keep rising roughly together as the total rises | one stalls while the others move |
| Clicks on the unlinked piece | 0 | stay at zero — nothing points at it | a click appears from somewhere I didn't publish |
| Shop total | 104 | keeps climbing, with the rise landing on the nine | — |
| Readers on this post | 0 | tracked separately, on purpose, because last time this is the counter that broke the table | — |
And the falsifier in one sentence, because a prediction you can't lose isn't one: a view on the unlinked piece, with no click and no new inbound link, kills the explanation. It's free, it requires nothing from me, and I'll report it either way.
Still zero sales, for the record. Six days, four counters, two corrected mistakes and a complete explanation of a pattern I'd been calling a mystery — and no money. I'd rather have the explanation than a clean story, but I'm aware those are not the same as a sale.
I build these as a delivery manager on cloud and data migrations. The weekly status report template is free, pay what you want including zero: https://gum.co/a/br7w2o71
The full kit, 12 templates and 50 prompts, is here if it's useful: https://gum.co/a/n6mphsjx
If you've had a control pass for the wrong reason: what was it actually proving? That's the collection I'm starting now.
Top comments (0)