Most scanners give you two states, and a pass can mean either “we looked and it was fine” or “we never looked”. Here is why we split them, and what it changed about the score.
Nearly every security scanner reports two states. Something passed, or something failed. It reads like a complete answer, and it is not, because a pass is doing two entirely different jobs at once.
Sometimes a pass means: we ran this check, there was something to read, and it came back clean. Other times it means: we could not run this check at all, so there is nothing to report, so here is a pass. Those two outcomes look identical on the screen and they could not be further apart in what they tell you.
Four ways we watched it happen
A shallow clone that only fetched the most recent commit found no secrets — because the secret was committed and deleted six months ago, and the clone could not see that far back. Green tick.
A repository with no lockfile produced no dependency vulnerabilities, because there was no resolved dependency tree to check against. Green tick.
A scanner that crashed halfway through returned partial output, which was parsed as a complete result with no findings. Green tick, and a good score.
A project whose database rules live in a hosting dashboard rather than in committed migrations was never checked for row-level security at all. The report did not mention it. Green tick, on the single most likely way that app could leak.
Each of those is the same bug wearing a different coat: absence of evidence rendered as evidence of absence. And each of them ends the same way — a green check, on a card someone posts publicly, for an app that is not safe.
So we report three
Found, checked and clean, and could not check. The third state is not an error state and it is not a footnote. It appears in the same place in the report that a result would have appeared, it says what evidence was missing, and it tells you where to look for that evidence yourself.
This costs us something. A report with four unchecked items looks worse than a competitor's report with four passes, even when the competitor never ran those checks either. Scores read lower. That is a real commercial cost and we are choosing to pay it, because the alternative is being the tool that told someone they were fine.
There is one rule underneath all of this, and it is the question we now ask before adding any check at all: what does this say when it did not actually run? If the answer is anything other than “it says it did not run”, the check is not finished.