Nullify exists to automate security work end to end — find, triage, plan, fix — and to do it with AI you can actually control and measure, so a team's security outcomes stop scaling with its headcount. That last part is getting harder for a reason that has nothing to do with us: attackers are scaling their own capability on the same cheap, open-weight models everyone else has, and the volume of AI-authored code moving through every codebase is climbing with it. If keeping pace means a human has to read and vouch for every AI-written fix by hand, you've just moved the bottleneck from writing code to reviewing it, and you're back to hiring reviewers in proportion to the problem.
The way out is not more AI-generated code. It's AI-generated fixes you can prove are correct, cheaply and automatically, so the human only steps in when proof genuinely isn't available. This post is about how we prove one important class of fix: re-run the exact exploit that triggered the finding against the patched code and watch it fail, then run the feature's own tests and watch them pass. That's a causal experiment about what the patched code now does, and where it applies it lets a security engineer accept an AI-written PR without becoming the machine's proof-checker. Below is where the experiment applies, how we've built it, and — just as importantly — where it doesn't apply and what we do instead.
We've argued the underlying idea before in Trust Issues: Can We Actually Validate Automated Vulnerability Repair?, which frames the "causal exploit check" as the contract a fix must satisfy before you trust it. That post is about validating repair in general. This one is narrower: exploit replay as the acceptance test for an autofix PR — the gate that, where it runs, lets a security engineer accept AI-written code with less manual review, not more. Today "where it runs" means our evaluation harness and one gated SAST integration test, not yet a blocking gate on customer autofix PRs — a scope we spell out below. It's the same instinct that runs our second-opinion review of authorization exploits before a finding ever reaches a human — doubt your own successes and demand proof — applied one stage later, to the fix instead of the finding.
A lot of "fix validation" in this space is really pattern matching on the patch. Does the diff contain the string Forbidden? Does it call the authorization helper? Did the fix touch the file we expected it to touch? These checks have a real place, and we use them ourselves in some paths, but they answer a weaker question: they tell you the patch resembles a correct fix, not that the vulnerability is closed.
We have both styles living side by side in our own benchmark corpus, which makes the contrast concrete. One of our pentest autofix cases verifies an IDOR fix by grepping the patched source: grep -q 'GetUserIDFromContext', a check for 403/Forbidden strings, an awk scope check that the auth call lands inside the right function. The acceptance criterion in that case's test.json is literally a field named mustContain: ["userID", "Forbidden", "Unauthorized"]. That's a token-level heuristic, and a fix that adds the right strings in the wrong place, or guards the wrong request path, sails straight through it.
Causal verification asks for something the grep can't give you: the actual attack request, replayed against the patched app, coming back empty-handed.
The mechanism we've built lives in our evaluation harness. For a benchmark case that ships with a runnable base repository, the verifier (verifyExploitNeutralized) asserts two independent things after the autofix agent has produced its patch:
test.sh exits 0. That script runs the upstream project's own unit tests plus a set of feature assertions against the live application — the endpoints that are supposed to keep working. As one test case's own comment puts it, these tests "exercise the legitimate feature contract independently of any LLM-written assertions. Any autofix that breaks the repo's own tests is a feature regression."The subtlety worth dwelling on is how one script run proves two independent things. Let the exploit's own success or failure bleed into the script's exit code and you can no longer tell a broken feature apart from a neutralized exploit. So the contract deliberately separates them: the exit code carries the feature-regression signal only, and the exploit's outcome is read separately from stdout. One of our test scripts spells out the rule in its header:
`` # Exit code signals feature regression ONLY (exit 1 = feature broken). # Exploit outcome is NOT encoded in exit code - it must be validated from stdout. # Feature assertions must use ... || exit 1 to propagate failures. # Exploit invocation must use bash exploit.sh || true so a patched-out exploit # (which the autofix harness relies on happening) cannot flip the exit code. ``
That || true on the exploit invocation is the whole trick. It lets the exploit fail — the outcome we're hoping for — without that failure masquerading as a broken feature. Feature assertions, meanwhile, are each gated with || exit 1 so any real regression aborts loudly. One run yields two clean signals that never cross.
Why insist on both halves rather than just the exploit? Because a fix that deletes the feature also closes the vulnerability. The most trivial "patch" for an injection bug is to rip out the endpoint that's injectable. We've watched a real autofix agent correctly remediate a credential-leak finding while silently dropping an unrelated field from an API response — a regression the exploit replay alone would never have caught, but the feature assertions did. Prove the exploit is gone and prove the feature survived, or you've proven nothing worth shipping.
A causal check is only as good as its oracle. Ours works by planting a flag — the literal string NU11IFI3D — into a database row or a /app/secrets.txt file in each target application, somewhere no legitimate route ever reads. The injected vulnerability is what makes that flag reachable. The design comment on one of our target repos states the causal dependency in a single line:
> No baseline controller reads this table — vuln.patch MUST introduce a route that explicitly queries cluster_secrets for the exploit to succeed. This preserves causal dependency (revert patch ⇒ flag unreachable).
That one sentence is the whole thesis. Because the flag is reachable only through the planted flaw, capturing it proves the flaw is live, and failing to capture it after a patch proves the flaw is closed rather than merely quiet. Revert the fix and the exploit fires again; apply the fix and it goes dark.
We'll be honest about a current limitation rather than sell the mechanism as airtight. Today the validators accept flag-in-stdout or a side-effect (the secrets file was modified, a table was dropped) as evidence of exploitation. Our own internal audit flags this as a soft spot — side-channel signals can false-positive, and a per-repo validator can fall back to the wrong repo's logic — and the direction we want to move is FLAG-only: require every exploit to actually print the captured flag, and fail closed when no real validator exists. The mechanism is strong and getting stronger; it isn't finished.
Here is where efficiency stops being a footnote. Replaying an exploit is not free. It means standing up a Docker Compose stack, seeding a database, driving real HTTP traffic, and letting an LLM-authored exploit script run — minutes of wall-clock and real compute per case. The field that turns this on is disabled by default for exactly that reason; the schema doc says it plainly: "Disabled by default because it requires Docker and adds minutes per test case." It only runs at all for cases that have a runnable base repo with a registered validator.
That's a deliberate spend decision. Spending minutes and dollars to prove a fix is worth it precisely when the alternative is a security engineer spending their scarce attention to guess at one. The whole point of the find → triage → program-manage → fix loop is to buy down human effort and needless token spend at every step, and exploit replay does that at the fix step: it converts the most expensive review there is — a person's — into an automated experiment, and reserves that experiment for the findings where a runnable attack exists to make it decisive. Where the experiment can't run, we don't burn the compute pretending it can.
That selectivity is why we treat this as production-adjacent rather than universal. The verification is real and wired end-to-end: at least one of our SAST autofix engines runs it in an integration test that drives the real fix agent, with real LLM calls, against live path-traversal cases across three languages, asserting both that the exploit is neutralized and the feature preserved. That test sits behind an explicit build tag and a 30-minute timeout — an opt-in, expensive regression check rather than part of the fast CI loop, and not yet a blanket blocking gate on every customer-facing autofix. Our internal design work is candid that turning this into a default acceptance gate across all autofix paths is a near-term goal, not a solved fact. We'd rather state the roadmap than imply coverage we don't have.
This is the honest boundary of the technique, and it deserves to be drawn sharply. Exploit replay works when there's an exploit to replay: DAST-reachable, benchmark-validated code-CWE classes with a base app you can attack. Large parts of the problem don't fit that shape, and we don't pretend they do.
SCA findings — "your dependency has a known CVE" — have no attack script to re-run, so the proof strategy changes to match the artifact:
1.2.0 finding has a same-major fix at 1.2.6, so "fixing" it by rolling back to 0.2.4 is exactly the regression the guard is built to reject).DAST autofix, even though the finding was born from a live attack, currently verifies fixes at the source-diff level — cloning the repo and running a fix agent against source — rather than re-attacking a redeployed target. We're naming that limitation, not hiding it: the "re-run the same HTTP exploit against the deployed app" loop is where the benchmark harness leads, but it isn't yet the production DAST gate.
The through-line across all of these is one instinct — prove it, don't just diff it — applied to whatever proof the finding actually admits. Sometimes that's a replayed exploit, sometimes a re-resolved lockfile or a settled CI run. The bar stays the same height; only the mechanism changes to fit what's provable.
None of this is a black box, and none of it takes the wheel out of your hands.
Every autofix outcome resolves to a typed, queryable verdict — fixable, not-fixable, or admin-set — that's persisted and surfaced through the API and CLI. We draw the not-fixable line deliberately: a finding is marked not fixable only when the fix agent itself returns a structured failure reason, meaning we genuinely tried and this can't be auto-fixed. Infrastructure or transport errors on our side stay as retryable errors, so a hiccup in our own plumbing never gets blamed on the finding. Every failure is typed and attributable back to its real cause. On the SAST path, we persist LLM cost, token count, and duration for each autofix to both DynamoDB and Postgres, so the spend that buys a verified fix is itself measurable. (One current limit worth naming: those metrics are last-write-wins per iteration today, with full per-iteration tracking still on the list — visibility we're deepening, not claiming as complete.)
And the human lever is always there. SCA autofix PRs stay in draft until CI settles or the iteration budget is spent; the system never silently merges, and once the agent's budget runs out it explicitly hands off to a person. Operators have a fully manual override path to clone, hand-edit, and upload a fix of their own. Our detection CLI is built around the same posture — AI proposes, a human can review, select, or edit any artifact between phases — as a first-class workflow: findings move through discrete scan → triage → upload phases with a per-finding artifact file persisted at each step, and the upload step exposes per-finding prompts (--interactive) alongside selection and confirmation gates (--select-all, --auto-confirm). Autonomous efficiency is the default, and the override is one command away whenever you want it. That posture is the whole point: automation that earns trust by proving its work, with a security engineer holding a lever they can pull at any time — neither "trust us, it's fully autonomous" nor yet another review queue.
An autofix PR is a claim: this vulnerability is gone and your feature still works. Diff-similarity checks let you hope that claim is true. Exploit replay lets you prove it — by re-running the exact attack and watching it fail, then re-running the feature's tests and watching them pass. Where a runnable exploit exists, that proof is the strongest acceptance test we know of, and where it runs it's what lets a team accept AI-generated fixes with less manual review rather than more — proven today in our evaluation harness and one gated SAST integration test, with wiring it into the customer autofix path as the near-term goal. Where it doesn't exist, we don't fake it: we prove whatever the finding admits — a re-resolved lockfile, a settled CI run — and make every outcome a verdict you can audit.
Automating the fix was always the easy half. The hard half, and the one that decides whether a security team can keep pace without hiring a reviewer for every AI-written PR, is automating the proof that the fix is real.