Discovery is effectively infinite now. Cheap, capable open-weight models have collapsed the cost of generating findings: a self-hostable model like GLM-5.2 can now surface plausible vulnerability candidates for cents apiece, at a recall that would have required a commercial scanner not long ago. That cuts both ways. The same economics that let a defender scan every commit continuously let an attacker — or an automated tool chasing a big number — generate a flood of candidate findings faster than any human queue can absorb. When producing findings is nearly free, the scarce thing is no longer detection. It is knowing which findings represent real risk, and getting that risk out the door. So the metric that matters is risk actually removed, not how fast you triaged the flood.
Ask an AppSec team how their program is doing and you will almost always get a time instead. Mean time to triage. Mean time to detect. Mean time to remediate. The whole industry has settled on speed as the headline number, and it is easy to see why: a duration is legible. It fits on a slide, it trends over a quarter, and everyone already knows whether smaller is better.
To be fair, time is not always a vanity metric. Plenty of programs owe a genuine, time-bound duty of care: PCI DSS, FedRAMP, and many contractual and regulatory regimes impose real remediation-window SLAs, and "critical vulnerabilities patched within N days" is a legitimate obligation you have to be able to prove. Our argument is not that those clocks should stop mattering. It is that a remediation window is a floor you must clear, not the thing that tells you your program is actually reducing risk — and treating triage speed as the headline score quietly substitutes the floor for the goal.
The problem is that a stopwatch does not know whether you ran in the right direction. A metric that rewards closing a ticket quickly rewards a fast wrong answer exactly as much as a fast right one. Mark a finding as a false positive and the clock stops; triage something out with a confident, decisive, and completely mistaken verdict, and the clock stops just the same. The scoreboard lights up green either way.
We want to make a sharper claim than "metrics can be gamed." We think time-to-triage is measuring the wrong thing entirely. The metric that matters is risk actually removed per unit of human and AI effort spent — outcomes, not activity. And we know this reframe has teeth, because we have watched the speed number lie inside our own product, and we have already had to build the machinery to catch it.
Start with a design fact from our own schema. In our findings model, is_resolved is a computed expression: a finding counts as resolved if it was superseded by a newer scan, or allowlisted, or archived, or marked a false positive, or marked a duplicate, or has a fix timestamp. Six different states, one tick in the "resolved" column.
That is not a bug. Each of those states genuinely means "stop showing me this." But notice what it does to any speed metric built on top: a finding that was correctly identified as noise and a finding that was actually fixed become indistinguishable in the count, and therefore in the average time to reach it. The fast right answer and the fast wrong answer settle into the same bucket. If your scoreboard is "how quickly did findings reach a resolved state," you are rewarding both.
We learned how much that matters the hard way. One of our own post-mortems, half-joking, was titled "The Case of the 40,000 Missing False Positives." A view meant to surface everything we had triaged out as noise — the single best value story an AppSec platform tells, "look how much we filtered so your engineers never saw it" — was built on a filter that asked for false positives that were not resolved. Because a false positive is resolved by definition, that set was empty by construction. At one customer, close to 40,000 correctly-triaged-out findings were sitting invisible in the UI. The activity had happened. The number that should have represented it was structurally incapable of showing it. The closing line of that write-up has stuck with us: the prettiest first explanation is often the wrong one, and "it compiles" is not the same as "it's correct."
Speed metrics fail in a second, more embarrassing way: they conflate stories that have nothing to do with each other.
When we were building a new metrics API, an engineer found a customer finding whose triage, by the database timestamps, took six months — dragging that customer's mean-time-to-detect out to roughly 47 days. Nobody had spent 47 days triaging anything. The commit that introduced the vulnerability simply was not scanned until much later, so "detection time" was silently measuring scan lag, not our latency. A colleague put the real problem plainly: the number customers actually care about is how long the vulnerability lived in production, and that is a different metric from how long it took us to get to it, which is different again from how long it existed before we were ever onboarded. One "time to detect" field, three unrelated stories folded into it.
Our answer at the time was defensive surgery — Tukey fences, IQR-cleaned aggregates, medians over means — the same outlier-scrubbing logic now living in both our Go metrics code and the SQL behind our reporting endpoint. It works. But step back and notice what it tells you: keeping a single speed number merely honest takes that much machinery. We have a regression test guarding against a real production incident where an integer overflow made mean-time-to-detect-and-triage render as roughly 213 billion days. The headline metric displayed pure nonsense, and every dashboard downstream of it still looked fine. The speed number is fragile, easy to corrupt, and its corruption is invisible precisely because everyone trusts it as the summary.
The deepest version of the problem is that speed and correctness are genuinely orthogonal, and any honest system has to track them as two axes. Ours does. Our triage agent produces two separate confidence scores for every finding:
confidence — how sure we are the finding is real and exploitable.assessment_confidence — how sure we are the triage decision itself is well-supported.These are deliberately independent. From our own rubric: "A finding can have confidence: impossible and assessment_confidence: high when exact counterevidence proves it is junk." In other words, the system can be completely decisive — a fast, high-confidence call — while the answer to the underlying question is "this is not real." Collapsing those two axes into "we triaged it quickly" throws away exactly the information that tells you whether the speed was worth anything.
Our triage prompts spend most of their guardrails fighting the manufactured-confidence failure mode directly: "Do not treat 'not found yet' as counterevidence. Treat it as uncertainty." "Do not create a final attack story by assuming the missing system is vulnerable." Those rules exist because the easiest way to make a triage queue fast is to let it guess confidently under pressure — which is the industry-wide bad habit the speed scoreboard actively incentivizes.
Here is the reframe with teeth, and the reason we can make it without hand-waving: the data model for the right scoreboard already exists in our product. It is just under-narrated.
Sitting in the same reporting query as the whole mean-time-to-everything family, we already compute:
We are building that reframe straight into the campaign view. Mean-time-to-remediate becomes one tile among six or seven: detected, patch-ready, resolved, and backlog counts form a funnel; time saved and story points get their own visually separated "Impact" cluster; coverage and merge-ready percentage round it out. That impact-cluster scoreboard is a near-term capability — expect to see it land on the dashboard soon rather than a metrics page led by a stopwatch. It is what a scoreboard looks like when it is built around outcomes.
We will be self-critical here, because the reframe demands it: even our own external docs still list MTTR first under metrics and reporting, ahead of story points and risk reduction. Speed is legible in a way "exploitability-weighted risk reduction rate" is not, and that legibility is exactly why the whole industry — us included, until we corrected it — keeps defaulting to the easy wrong number.
Two principles fall directly out of putting effort in the denominator.
First, spend where it earns an outcome. We are not trying to triage every finding for the same flat cost. Most findings are triaged out for one to five cents because that is all the evidence they require. A finding that keeps proving exploitable earns a deeper, more expensive impact analysis — and even a five-to-ten-dollar deep dive is far cheaper than putting a human security engineer on it. A flat per-finding budget optimizes throughput; variable spend tied to demonstrated risk optimizes the ratio we actually care about.
Second, do not pay twice for the same outcome. In one cost investigation, more than a quarter of a customer's triaged findings turned out to be pure duplicates — the same underlying issue re-triaged from scratch on every scan because a dedup key was broken. That is textbook activity-metric hollowness: findings-processed-per-day looked healthy while a large fraction of the spend bought nothing. The same logic drives how we schedule expensive work: rather than eagerly generating an autofix on every possible trigger, our planning layer decides when a fix is actually needed, so an outcome gets paid for roughly once instead of every time something twitches.
None of this means removing the human. The point of an outcome-shaped scoreboard is not to run unattended — it is to make the levers legible so a security engineer can pull them. Leaders set the scope, the automation level, the deadlines, and the success criteria per campaign; PR-gate controls ship off by default so the system cannot self-approve into a merge until a human opts it up; and every phase of our review workflow produces hand-editable artifacts that the next phase strictly re-validates. Because every triage verdict carries its two confidence scores, its cost, and its trace, an engineer who distrusts a call can open it up and see exactly why the system decided what it decided. That auditability is what makes the reframe usable instead of just another opaque score to trust.
We have made this argument before in a narrower form. When we asked whether automated vulnerability repair can actually be trusted, the honest metric we landed on was not "the scanner stopped complaining" — a fast, cheap, and frequently wrong signal — but the merge-ready rate: the share of fix PRs a team merges with no further edits and no back-and-forth. That is an outcome. It is only earned when the fix is real, the context is honest, and the volume was kept to genuine risk. Triage speed is the same story one step upstream. A Director of Product Security we spoke with had independently reached the identical conclusion and written it up under the banner of impact versus triage volume and speed — the "fix it faster" reflex, he argued, answers the wrong question.
Improving triage was never the finish line. It solved a small part of the product-security value chain — and mostly it just made the rest of the problem more visible. Discovery is effectively infinite now, as the opening said; the constraint has moved to what you do about it, at what cost. So measure that. Not how fast the queue emptied, but how much real risk left the building, and how much human and AI effort it took to get it out the door. Everything else is a stopwatch timing the wrong race — which is the same complaint we make from the other side, about raw finding counts, in The Vulnerability-Count Arms Race Is a Vanity Metric.