Every security leader is being asked the same question this year: prove the program is working. The headcount to answer it is flat, the attack surface is not, and the number most teams reach for — how many vulnerabilities we found, or how many we closed — is the one number that no longer means anything. That is the actual stakes here. Not "our dashboards are noisy," but "the metric you are using to justify budget, staffing, and board-level confidence has quietly stopped measuring whether you are in control."
It stopped meaning something because raw finding count just became trivial to manufacture on both sides of the fight. Open-weight models you can download and run yourself are now good enough at vulnerability detection to rival flagship closed coding agents that cost orders of magnitude more — and they run for a rounding error in token spend. Read that as an attacker and it means exploit attempts, at volume, for pennies. Read it as a defender and it means findings — real ones, plausible-looking false ones, indistinguishable in a raw count — can be generated faster than any human can read them. When both sides can print findings by the thousand for the price of a coffee, "we surfaced 1,296 vulnerabilities" is not a measure of thoroughness. It is a measure of how cheap generation got. The arms race is real; the count is the vanity metric it produces.
So we want to argue something uncomfortable, including for us: more findings found is not the same as more risk removed, and "findings closed" is barely better as a proxy. A vendor that reports "we surfaced 1,296 vulnerabilities" sounds more thorough than one that reports 137 — even when 137 is the honest number and the other 1,159 are noise. In a world where a cheap, downloadable model can inflate that first number on demand, the only figure a leader can defend is verified risk actually removed. We can show what the gap between the two looks like with real numbers from real customer codebases.
Start with a concrete measurement. On a real 31,000-file customer codebase, four of our own SAST rules produced 1,296 raw findings. We tuned those four rules — pattern-level exclusions, path-based exclusions for vendored and minified code and test projects, and one structural change that ripped a blanket request-handler regex match out of a ReDoS rule entirely — and re-measured against the same code. The result:
PasswordHelperThat is 1,296 raw findings collapsing to 137, an 89.4% reduction, with all 8 true positives preserved. Read the denominator again: across those four rules, 8 findings represented real risk. A vendor optimizing for "we found 1,296 vulnerabilities" is reporting a number that is over 99% noise relative to the 8 that mattered. The count went down by an order of magnitude and the risk coverage did not change at all — every genuine issue survived the cut. If raw count measured risk, that would be impossible.
This is not an isolated bad rule. Our fleet-wide false-positive audit ranks roughly a hundred rules by noise, and a single JavaScript regex rule alone accounts for over 22,000 raw hits across the fleet. The "vulnerability count" that an untuned scanner reports is, structurally, a number dominated by a handful of pathologically noisy patterns. It measures how loud your detectors are, not how exposed you are — and cheap AI generation only makes it louder.
The cost of getting this wrong is not abstract. We have written before about how false positives are how AppSec teams lose their developers — the moment engineers learn that most of what a tool flags is junk, they stop reading any of it. A count-maximizing tool doesn't just misreport risk; it actively burns the trust it needs to get real fixes shipped.
Deduplicating and tuning noise is only half the story. The other half is that even distinct, technically-real findings differ enormously in whether they represent exploitable risk. This is where counting falls apart most completely, and it is squarely a program-management problem: deciding what to page a human about.
Consider a single Software Composition Analysis finding we worked through: a Go 1.24.1 dependency in a customer service that bundled 32 CVEs against the same package version. Today a naive scanner takes the maximum severity across all 32 and pages the on-call engineer on Critical. But when we run per-CVE reachability analysis from the application's actual entrypoints, the picture inverts:
Of 32 "Critical" CVEs, 3 are confirmed reachable. In the specific example that motivated this work, every Critical CVE in the bundle was for an encoding/asn1 code path the application never entered, while the only vulnerable function it actually reached was a Medium-severity bug in net/http. The raw count says 32 Criticals demanding urgent attention. The verified answer says: 3 reachable, and the thing that should actually change your priority is a Medium. A program managed by count would have paged an engineer at 2am for 21 vulnerabilities in code that does not run. (We go deep on this pipeline in Per-CVE Reachability: Doing the Triage Without the Headcount.)
The mechanism here matters, because it is where efficiency stops being a footnote and becomes the whole design. The reachability pipeline reads deterministic facts first — path classification, dependency-tree depth, whether a package is even imported — from a precomputed dossier. "This package is not imported at all" is decided with zero LLM calls. Only genuinely ambiguous cases escalate to an AI subagent, and that subagent runs on a cheap model under a hard budget: a dollar ceiling of a few cents of token spend per finding, per-tool call ceilings that scale with the number of CVEs in the bundle, and a timeout that starts at 45 seconds and grows with that bundle size. We batch all of a finding's CVEs into one LLM call because a 32-CVE bundle shares almost all of its context — only the affected-symbol identity differs — which saves roughly 30x the prompt cost versus analyzing each CVE separately. Purpose-built tools do the rest: one derives the vulnerable symbols for a CVE straight from its advisory, one finds the in-app callers of a symbol with vendored and test callers already filtered out, and a call-hierarchy primitive walks the graph — together keeping a typical reachability decision to a handful of tool calls where the same task takes 15–20 without them. Getting the count right is cheaper than getting it large, if you design for it. That asymmetry is the whole answer to the arms race: if attackers can generate findings for pennies, the defender who wins is the one who can verify them for less, not the one who counts louder.
Three guardrails keep this from becoming its own vanity trick — the mirror-image failure where a reachability filter quietly buries findings you needed. First, the classifier can only ever lower priority — "no upgrades, the classifier's draft is the ceiling" — and any finding whose reachability is genuinely unknown counts as reached for the severity computation, because a broken analysis "must never under-prioritize a real exploit." Second, a known-exploited-vulnerability floor: if a CVE sits on CISA's KEV list, static symbol reachability — which routinely misses reflection, dynamic dispatch, and build-time codegen — is never trusted enough to crush it to Informational. The strongest demotion a KEV can take is a single band, so a vulnerability that is being exploited in the wild stays visible no matter what the reachability evidence says. That rule is not a slide; it is enforced in EffectiveSeverity and locked in by unit tests. Third, the entire path sits behind one feature flag per stack and defaults off: a security engineer who wants the raw, unfiltered severity keeps it; a team that trusts the analysis flips one lever. And when any component fails — a parse error, an AI timeout, malformed output, a missing metadata source — the failure mode is always "no severity change," never "we silently suppressed a finding because our analysis broke." You cannot lose a real risk to an infrastructure hiccup.
The strongest evidence that we mean this is that our own evaluation framework refuses to reward raw counts — the same discipline we are asking the industry to adopt is wired into how we measure ourselves. It has to be: in a world where a cheap model can emit findings by the thousand, an eval that scores on volume is an eval that rewards the arms race instead of the outcome.
Our benchmark corpus pairs every vulnerable code sample with a test.json of expected findings, and our detection scoring does 1-to-1 matching between scanner output and that ground truth. This has a deliberate consequence: if a scanner reports two findings at the same line where the truth is one real bug, the second finding does not silently collapse into the first — it surfaces as unexpected and fails the evaluation. You cannot pad your numbers by double-reporting the same vulnerability. Scoring is expressed as precision and false-positive rate; raw finding count is explicitly discarded as a metric. And recall is severity-weighted (a Critical counts 4x a Low), so two scanners that both "find 10 things" are not equal if one missed a Critical and the other missed a Low. Per-CWE tracking means good aggregate numbers cannot hide a systematic blind spot in one vulnerability class.
The cleanest expression of "verified risk reduction" lives in how we score autofixes. A fix only counts as fixed if three things hold, checked mechanically:
"Fix rate," in our benchmarks, is not a count of tickets closed. It is the count of vulnerabilities that were re-scanned and confirmed removed with no regressions. That is the literal machinery of verified risk reduction, written as working code. We also decline to attempt fixes we cannot responsibly automate — architectural-effort findings and false positives are skipped — so we cannot inflate a "fixed" number with unverifiable attempts. The benchmark README states the goal plainly: balance burning down security risk against the friction imposed on developers. Not "how many findings." Risk removed per unit of developer friction.
That same anti-noise principle runs through our decision-making benchmarks. An agent can fail for doing too much — too many tickets, too many comments — just as easily as for doing too little. Excess PR comments carry an explicit negative weight in the score; certain over-eager actions are hard failures regardless of whether the underlying finding was real. Noise is not neutral in our scoring. It is penalized.
We would be doing exactly what we are criticizing if we told you "just count resolved findings instead." So here is the uncomfortable nuance, stated plainly.
Our production metrics API deliberately goes beyond a raw number: it exposes resolved counts, backlog counts, allowlisted counts, in-progress counts, findings with an open autofix PR, an effort-weighted backlog in story points, and ten separate mean-and-median timing fields for detection, triage, PR creation, review, and merge. But "resolved" in that schema is a union: a finding is resolved if it was superseded, allowlisted, archived, marked a false positive, deduplicated, or actually fixed in code. A finding correctly triaged away as noise counts the same as one genuinely repaired.
That is the right design — suppressing noise is a legitimate outcome, not cheating, and a program that filters 40,000 false positives away from its developers has done real work. In one case, a customer had close to 40,000 triaged-out false positives that a dashboard bug had hidden from view; that filtering was, as one of our engineers put it, "the single best value story we tell," quietly invisible. But "resolved" and "fixed" are not the same word, and "we resolved 90% of findings" would be a brand-new vanity metric if it were quietly mostly-allowlisting dressed up as remediation. The honest thing is to keep the distinction visible — resolved ⊇ {fixed, allowlisted, deduped, marked-FP, archived, superseded} — and never let the aggregate launder the difference.
The counting logic can fail in the other direction too — undercounting real risk. Our own CSPM finding dedup once collapsed many distinct cloud resources into a single finding: the natural key is {ruleID}#{resourceARN}, and a class of account-scoped and multi-resource rules was emitting an empty ResourceARN, so every resource those rules flagged degenerated to the identical key {ruleID}# and folded into one row. Dozens of genuinely separate misconfigurations, reported as one. The fix was to synthesize a stable per-resource identifier when a rule supplies none — cspm:<provider>:<type>:<region>:<id> — plus a reconcile backstop that deterministically disambiguates (and warns about) any findings that still collide, so no rule can silently merge distinct resources again. That is the point: the fix for count-as-vanity-metric is not "always report fewer findings." It is "get the counting logic correct against ground truth." Sometimes correct means far fewer. Sometimes it means catching things a broken dedup key was hiding. The only durable answer is to anchor every number to verified reality — which is exactly why our production timing metrics use outlier removal so one stale multi-week finding can't distort a reported average, and why we feed real production findings back into the benchmark corpus, verified against the live detector, before we ever call a vulnerability class "covered."
The whole find → triage → program-manage → fix loop only works if the numbers at the end mean something. A tuned SAST ruleset that takes 1,296 findings to 137 while preserving all 8 real ones. A 32-CVE bundle that reachability analysis resolves to 3 things worth paging on, for pennies of token spend. An autofix that only counts once a re-scan proves the vulnerability is gone and nothing new broke. None of these make the count bigger. Every one of them makes the number more honest — and honest is what a security leader can actually act on.
Internally, one of our own put it about as directly as it can be put: who cares how many hypotheses we generate — the metric people care about is how many exploits were validated. That is the entire arms-race lesson in one sentence. When generation is nearly free — a rounding error in token spend, on a model anyone can download — raw count is the easy number to report and the wrong number to optimize, because either side can inflate it at will. Verified risk reduction is harder to measure, harder to fake, and the only one that corresponds to your organization being safer than it was yesterday. We would rather report the honest, smaller number — and show you the re-scan that proves it.
It's the same bad incentive we've caught in our own reporting from a different angle: a program that optimizes for closing findings fast is just as easy to game as one that optimizes for finding a lot of them, and we go into that failure mode directly in Triage Speed Is the Wrong Scoreboard.