A Test Report Should Say What It Cannot Prove

A test report can be accurate and still lead to the wrong decision.

The problem is often the space between what the test checked and what the reader thinks it checked. "Passed" is a short word. It can quietly absorb claims about usability, reliability and readiness that were never part of the test.

In my game projects, Instinct does much of the implementation and testing. I remain responsible for the direction and the decision to ship. That makes the handoff report part of the product work. I need to know what the evidence supports before deciding what should happen next.

I have written about stop rules before. This is the reporting side of the same problem: how to describe a result without letting it become larger than the observation behind it.

Start with the claim

"The feature works" is too broad to test usefully.

"A hostile shot cannot award the same near-miss reward twice" is narrower. So is "the intended gameplay screen appears after launch." Neither statement proves that someone understands the mechanic or enjoys playing it.

Those are different questions. They need different evidence.

Before reading a result, I want to see the claim in a sentence. If the sentence contains several claims, it probably needs to be split. A test should not inherit a bigger job because the report uses a convenient label.

Put the observation beside it

A useful report connects a claim to an observation someone can inspect.

For a hypothetical game build, that might mean a regression trace showing that one projectile cannot award twice. For a visual claim, it might mean a capture of the relevant screen. For a touch-input claim, it requires evidence from the target interaction, not a desktop approximation quietly presented as the same thing.

These are examples of evidence requirements, not a claim that a current candidate has passed them.

The distinction matters because one artifact can be real but irrelevant. A screenshot proves that something was rendered. It does not, by itself, prove that it was the right screen, the right build or the state the task required.

Add the limit while the result is still fresh

I would put a short limit beside every meaningful result:

• Claim: the same projectile cannot award the reward twice.
• Evidence: the regression route checks repeated contact with that projectile.
• Limit: this does not establish that the reward feels fair on a phone.
• Next question: can a player understand why the reward appeared?

The limit is not an apology. It stops a precise result from becoming a vague promise.

It also makes missing evidence visible. A device check that has not happened is not a failed device check. But it is not a pass, either. Reports need language for all of those states.

Keep the next step proportional

The report should make the next action easier to choose.

If the remaining question is about player understanding, another run of the same logic test will not answer it. The useful next step could be a clearer cue or a focused playtest. If the remaining question is about an incorrect screen, adding a feature is a distraction.

This is where fast AI-assisted work can help or hurt. Preparing another candidate is cheap enough that it can hide the fact that the missing evidence has not changed.

I do not want a longer report for its own sake. I want a report that prevents me from approving a different claim from the one that was tested.

The strongest handoff is often a modest one: this is what we observed, this is what it supports, and this is what we still do not know.

Related writing

• Fast AI Work Needs a Stop Rule
• A Feature Isn't Finished Until the Player Can Read It

How this article was prepared

Instinct helped draft this article from my published project writing. I remain responsible for what appears under my name.