Authorization flaws — IDOR, broken access control, tenant isolation gaps — used to be expensive to find at scale. They resist signature scanners because the bug isn't in the code shape; it's in the gap between what an endpoint returns and what the caller was supposed to be allowed to see. Finding one meant a human reasoning about intent. Finding a thousand meant a team.
That economics just changed for the attacker too. GLM-5.2 — a freely downloadable, MIT-licensed open-weight model released in mid-2026 — scored 39% F1 on IDOR and authorization-vulnerability detection at roughly $0.17 per vulnerability found, beating the reported scores of flagship closed coding agents on the same task. One person can now point a cheap, self-hosted model at an API and hunt access-control bugs at a volume that used to require a staffed offensive team. The capability to find these bugs is no longer scarce, and it is no longer on the defender's side alone.
That is exactly why Nullify's thesis — automate the security work end to end, with AI that is controlled and measurable, so security outcomes decouple from headcount — lands hardest right here. When the marginal cost of finding a candidate vulnerability collapses, the bottleneck moves entirely to trust: can you tell a real authorization violation from a 200 OK that means nothing, at the same scale, without drowning an engineer in false positives? Speed of discovery is now table stakes on both sides. The defensible edge is trustworthy, low-false-positive validation. This post is about the part of our pipeline built to earn that trust: the step where we doubt our own exploits before we report them.
We've written before about proving exploits work. In Exploit Validation: Proof, Not Guesses we argued that a finding should ship with a runnable, reproducible demonstration that the vulnerability is real — proof of exploitation, not a scanner's hunch. What follows is a different mechanism that runs after that. Once an agent has attempted an exploit and recorded the result, a stronger reasoning model re-reads the recorded evidence and is allowed to overturn the verdict. No new requests hit the target. This isn't more proof-gathering; it's a cross-examination of the proof we already have.
The distinction matters because the hardest false positives in authorization testing don't come from an exploit that failed to run. They come from an exploit that ran, returned a 200 OK, and still means nothing — because the response body said "access denied," or the "victim" resource was public all along, or the two users were, on inspection, the same person. A first pass optimized for speed and volume will call that a hit. Somebody has to be paid to doubt it. That somebody is the Exploit Validator subagent.
The Auth Matrix agent is our authorization-vulnerability hunter, built for APIs with thousands of endpoints. Its design philosophy is blunt: Python orchestrates, LLMs decide. Code is plumbing; the model does the judgment. The run moves through a sequence of stages, each spending compute where it earns its keep.
It starts by inferring the world. Before any attack, we infer the target's privilege hierarchy and user states — roles like admin versus member, and states like banned, unverified, or restricted — from the credentials and personas the customer supplied at setup. These aren't hardcoded; they're discovered per target.
Then a grouping and analysis stage kicks in: a cheap, fast model clusters endpoints into logical resources, and a stronger model analyzes each cluster, ranks severity, and generates concrete hypotheses ("a member can read another org's records here") — the resource-mapping approach we cover in Mapping an API's Authorization Model Before You Attack It.
Next comes exploitation at scale. A fast model tests every hypothesis against the live target — potentially thousands of them across a large API. This is bulk, high-volume work, and it runs on the cheap tier deliberately.
When a repository is connected, successful exploits then pass through a white-box code-validation stage. For each apparent hit, a code-context subagent pulls the relevant source and traces the data flow — which handler serves the endpoint, where the authorization check lives (or doesn't), what the response is actually assembled from. That code context is attached to the exploit and later flows into the finding's write-up, so a confirmed vulnerability ships with the handler and data flow that produced it, not just an HTTP transcript.
Only after that does the validation stage run: a stronger reasoning model reviews the attempts the exploit phase reported as successful, weighs the recorded HTTP evidence against the inferred authorization model, and filters the false positives out. The survivors — and only the survivors — get cleaned up and written into findings a customer sees.
That validation stage is the subject of this post. Its job, in the words of the code's own docstring, is to "review each successful exploit and determine whether it represents a true authorization vulnerability or a false positive (e.g., intended public access, test endpoints, etc.)."
It would be cheaper to trust the exploiter. We don't — and not because the exploiter is sloppy. The fast model that runs the attack is already required to police itself: before it may call a finding successful, it has to supply differential proof. It must name a resource that belongs to the victim, show that the attacker accessed or modified that specific resource, and point to data in the response the attacker should never have seen. Absent all three, it's instructed to report the attempt as failed.
So the story is not "a careless model makes a mess and a careful one cleans up." It's stronger: even a careful, self-checking attempt gets independently re-litigated by a more expensive reviewer before it's allowed to reach a human. The validator reasons about three questions the first pass isn't equipped to weigh as deeply. Does the evidence actually support the claimed vulnerability? Does the response indicate real access, or an error dressed up as success? And, the one that matters most: does the access actually violate the expected authorization model, or is it access the target was designed to allow?
That third question is the whole point, and it's what separates this from a generic "re-run the curl and see." The validator isn't re-verifying that an HTTP call returned 200. It's asking whether a 200 crosses a boundary defined by the roles and user states inferred for this specific application. The validator is built to reason against exactly those inferred roles and states — a member versus an admin, a banned account versus a healthy one — so that crossing a boundary reads as a vulnerability while staying inside one that merely looks permissive does not.
The validator carries an explicit taxonomy of ways an apparent exploit turns out to be nothing, and it isn't hypothetical — customers have hit every one of these edges.
The most common trap is a status code that lies: a 200 whose body contains "access denied" or "permission denied," a 200 with {"data": null} or an empty list, a 200 carrying {"success": false}, or the soft 404 — a 200 wrapped around a "not found" message. Close behind are cases where the access was simply legitimate: attacker and victim are the same identity (self-access isn't a vulnerability), the endpoint is genuinely public, or the resource is shared with a team the user actually belongs to. Sometimes what looks like a breach is a security control working — a 429, a CAPTCHA, a WAF block, a temporary lockout — the system defending itself correctly. And sometimes the access is real but trivial (metadata-only, redacted or empty sensitive fields) or an API artifact entirely (an OPTIONS/HEAD preflight, a schema-introspection route, a health check).
Against that, the validator holds a matching set of true-positive signals it's built to keep: an attacker reading a victim's PII or payment data, a low-privilege user hitting an admin endpoint, a regular user acquiring admin capabilities, a banned or unverified user doing what their state forbids, one tenant reaching another tenant's data.
One customer report shows why a fixed ruleset can't do this job. They flagged an authorization finding as a false positive because "the authentication and authorization check is performed at the base controller" — the app enforced access centrally rather than per-endpoint, a pattern genuinely harder to infer than a simple per-route rule. That's precisely the application-specific logic the validator has to reason about, and precisely why reasoning against an inferred model beats a static checklist.
Reasoning against an inferred model has a sharp edge: sometimes the model isn't sure what the intended access even is. On one engagement a reviewer held back a set of IDOR-style findings because he genuinely couldn't tell whether every practitioner at an organization was meant to see every patient there. Without the authorization model as ground truth, neither a human nor a subagent can separate a real IDOR from intended shared access. The honest answer is "uncertain," and the pipeline is built to surface uncertainty as uncertainty — findings carry a confidence score and unknowns stay visible — rather than forcing every call into a confident yes or no.
The validator's confidence bands run from "clear vulnerability" down to "likely false positive, filter it out," with a middle band reserved for "uncertain, needs manual review." Its default direction is asymmetric by design. If the run hits its time budget before finishing, or the model never reaches an explicit decision on some exploit, that exploit is defaulted to filtered — with a recorded reason. The worst failure this stage can produce is a missed true positive that needs a re-run, never an unreviewed guess surfaced to a customer as fact. False positives waste the customer's scarcest resource; a re-run only wastes ours.
That conservatism has a real cost. A validator this cautious will occasionally reject a true finding when the underlying exploit attempt was weak — sent the wrong parameters, got a 400, and "proved" non-exploitability by testing almost nothing. We know this; it's an active area of work on the exploiter side (more recon, deeper effort, clearer definitions of what counts as an exploit). A downstream reviewer is only ever as good as the evidence it's handed. But given the choice, we'd rather re-run a scan than hand someone a finding that falls apart on inspection.
It's tempting to read "run a second, more expensive model over every hit" as a tax. It's the opposite — the economics of separating the two passes are the entire point. The cheap, fast model does the massive-scale probing: every hypothesis, potentially thousands, at a low per-request cost. The expensive reasoning model runs only on the small set that already came back successful, with a larger thinking budget spent where it counts. Spend cheap tokens on breadth; spend expensive tokens on the one question — "did this actually work, and does it actually matter" — that decides whether a human gets interrupted.
To make that concrete: in one internal run against a real target, the exploit phase flagged 70 successful exploits; after the stronger-model validator re-reviewed them, 5 were confirmed and 65 were filtered out. That's one run, not a universal ratio — the split shifts with the target and the quality of the underlying attempts — but it's a fair illustration of the shape. Roughly nineteen of every twenty "successes" in that run were noise that never reached a person. That is the efficiency thesis: the expensive step exists so the truly expensive resource — a security engineer's attention — is spent almost entirely on real vulnerabilities.
The validator reviews one exploit at a time. That's a deliberate choice: each finding gets the model's full attention, and there's no batch to skim past. Cost is optimized at the level of the whole pipeline, not by cutting corners on the last gate before a customer sees a result.
An autonomous filter that silently drops findings would be worse than no filter at all — you'd never know what you weren't seeing. So every decision this stage makes is recorded and inspectable.
Every validation decision carries a reasoning string and, when it filters, an explicit false-positive reason — recorded on the decision and captured in the run's trace, so a security engineer can read exactly why the model dropped a given exploit. The validator's intermediate reasoning is captured as discrete think-steps in the trace, and its running verdicts show up live in the transcript as "confirmed" or "filtered (false positive)" with a running count. And the whole run is traceable end to end: we record the surviving-exploit count on the trace span and pull the full cost and token breakdown for every run, tied to a trace URL. Every filtered finding is auditable back to the reasoning that filtered it.
The human lever is real without pretending the filter is optional theater. The validator's verdict is a keep-or-filter call — it never silently rewrites a finding's severity. When it thinks severity is off it can attach an optional suggestion — "upgrade" or "downgrade" — alongside its decision as a note for a human to weigh, not something the pipeline applies on its own. Combined with the recorded reasoning and confidence score, the default is autonomous efficiency, but the trace means a security engineer can always step in, check the model's work, and override it. That's the shape we want everywhere: the system does the tedious re-checking so you don't have to, and shows enough of its work that you can trust it — or catch it — when you choose to look.
A pentest report is only useful if you can act on it without re-verifying every line. The moment a customer has to independently re-check each finding, the autonomy has bought them nothing. One prospect put it plainly in their own success criteria: 100% of exploitable findings must ship with reproducible evidence — and internally, "validated authorization proofs-of-concept" is the capability we treat as both the hardest to get right and our biggest differentiator when we do.
Which brings the argument back to where it started. When an attacker can rent a $0.17-per-bug model to hunt authorization flaws at team scale, finding candidates stops being the hard part for anyone. The Exploit Validator is how we earn trust on the part that's still hard: deciding which candidates are real — cross-checking each against the authorization model of the specific application, defaulting to silence when the evidence doesn't hold up, and recording why. That's the work that turns a pile of maybe-vulnerabilities into a short list a human can act on. Getting the outcome right for the least human effort doesn't mean removing the human. It means making sure that when we do put something in front of one, we've already done the doubting.