Nullify

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

What Happens When an Autofix PR Fails CI, and Why That's Where the Real Work Starts

The walkthrough below is an illustrative example, not a customer PR.

A draft pull request opens against billing-service. It fixes an insecure direct object reference: the invoice endpoint returned any invoice by ID, and the change adds an ownership check before the lookup. Four minutes later CI goes red. test_invoice_download fails with a 403.

Most autofix tools stop here. The PR has a red X on it, a developer glances at it, and it joins the queue of bot PRs nobody owns.

An opened PR is a proposal. A merged PR is a fix. Everything between the two is the part that decides whether security work ships, and it's the part most tools leave to you.

Why do autofix PRs fail CI?

An autofix PR CI failure is rarely a sign the security fix is wrong. More often the fix is right and something around it disagrees.

The common causes:

  • A test encoded the vulnerable behavior. The fixture fetched an invoice as a user who didn't own it, because nobody thought of that as a bug when the test was written.
  • A dependency upgrade changed an API. The patched version renamed a function the code still calls.
  • A lint or type rule the generator didn't know about. The repository enforces a style the patch didn't follow.
  • The branch fell behind. Another merge changed the same file, and the PR no longer applies cleanly.

Each of these needs a different response. None of them means "throw the fix away."

Step 1: Read the failing log, not just the status

The first job is to find out which failure this is. A red check is a status. The log is the evidence.

In the example, the log says test_invoice_download expected 200 and got 403. The test logs in as user_b and downloads an invoice owned by user_a. That's the exact access pattern the fix was written to block.

The test isn't catching a regression. It's asserting the vulnerability.

So the correct change is to the test, not the fix: the fixture should download its own invoice, and a new assertion should confirm that a different user gets 403. That's one corrective commit, pushed to the same branch, with a commit message that says why.

Restraint matters here. If the log points at the fix itself, for example a null check that crashes on a missing owner field, the fix gets revised. If it points somewhere the fixer can't safely reason about, the effort stops and a human is told why. Repair has a budget. It isn't a loop that runs until CI happens to pass.

Step 2: Answer the reviewer in the PR, not in a ticket

CI goes green. A reviewer leaves a comment on the ownership check: "Why not do this in the middleware?"

That question is where bot PRs usually die. Nobody replies, the reviewer moves on, and the PR ages.

A fix that wants to merge answers inline. Either it explains the choice (the middleware doesn't load the invoice, so it can't check ownership without a second query) or it updates the code to the reviewer's approach. The conversation stays on the diff, where the context is.

Step 3: Rebase when the branch falls behind

Two days later someone merges a refactor that touches the same controller. The PR now conflicts.

Rebasing is dull work, which is exactly why it doesn't get done on PRs nobody owns. A fix PR that's behind main is a fix PR that won't merge. Keeping it current is part of the job, not a favor.

Step 4: Promote the PR and hand it to the right person

Only once CI is green does the PR move from draft to ready for review. Then it needs an owner who can approve it.

The right reviewer isn't "the repository." It's the engineer who most recently changed those lines, then whoever CODEOWNERS names for the path, then the owning team. That order puts the PR in front of the person with the most context, not the person with the most notifications.

The lifecycle at a glance

StageWhat can go wrongWhat a finished autofix does
Draft openedPatch misses the root causeIndependent review before the PR opens
CI failsTest asserts old behavior, API changed, lint ruleRead the log, push a corrective commit within a bounded budget
ReviewReviewer question goes unansweredReply inline or revise the code
Branch behindMerge conflictRebase
GreenNobody is assignedPromote to ready for review and assign the code owner
MergeAuto-merge skips human judgmentA human merges, always

What to check if your autofix tool stops at "PR opened"

You can test any tool, or your own harness, against this list:

  • Does it read CI logs, or just report pass/fail?
  • Does it push corrective commits to its own PR?
  • Does it have a stated limit on repair attempts?
  • Does it reply to review comments on the diff?
  • Does it rebase stale branches?
  • Does it keep the PR as a draft until CI passes?
  • Does it assign a specific reviewer, and how does it choose?
  • Does it refuse to merge on its own?

If most answers are "no," the tool writes patches. Your engineers still do remediation.

How Nullify works the PR after it opens

This lifecycle is how Nullify's Autofix is built. Before the PR opens, a separate review step checks that the change fixes the root cause, preserves behavior and stays minimal, and Nullify runs the scanner, build, test and exploit checks that the workflow has. Then it opens a draft PR.

After that, Nullify reads failing CI logs and pushes corrective commits within a bounded effort budget, replies to review comments inline, rebases when the branch falls behind, and moves the PR to ready for review once CI passes. Reviewers come from a ranked cascade: recent authors of the changed lines, then CODEOWNERS, then the owning team, with at most three reviewers per PR. Developers can mention Nullify on the PR to ask why or ask for a different approach.

When a fix isn't appropriate, Nullify falls back to fix guidance and a ticket with reproduction steps, tagged to the owner. Fixes only start from findings that Triage & Remediation has confirmed are real and worth fixing.

Nullify never merges. A human always makes the final call.

An opened PR is a proposal

The red X on billing-service wasn't a failed fix. It was a test asserting the bug. The work that mattered was reading the log, changing the test, answering the reviewer, and keeping the branch current.

An opened PR is a proposal. A merged PR is a fix.

Frequently asked questions

Why do AI-generated fix PRs fail tests?

Usually because something around the fix disagrees with it: a test that encoded the vulnerable behavior, a dependency API that changed in the patched version, a lint rule, or a branch that fell behind. A failing test doesn't automatically mean the security fix is wrong, so read the log before discarding it.

Should an autofix tool change tests to make CI pass?

Only when the test asserts the vulnerable behavior, and the commit should say so. Changing a test just to turn CI green hides regressions. A good fixer explains every test change in the commit message and the PR, so the reviewer can check the reasoning.

How many times should an autofix retry a failing build?

A fixed, small number. Repair should have a bounded effort budget, after which the PR is handed to a human with an explanation. An open-ended retry loop can wander away from the original fix and wastes CI minutes.

Should autofix PRs be auto-merged once CI passes?

No. Passing CI shows the change builds and tests pass, not that it's the right change for your system. Keep human approval on every security fix merge, and use automation for everything up to that point.