Building your own AI security harness looks like a fun weekend project. The cost shows up later, when your engineers are maintaining custom agents and broken wrappers instead of shipping features.
The weekend version is genuinely impressive. Pull findings from a scanner, send each one to a frontier model with some surrounding code, ask whether it is real, and open a pull request if it is. By Sunday night it triages a handful of findings and writes a clean fix.
That prototype is real. It is also the cheapest part of the system you have just committed to owning.
After more than three years of building security agents at Nullify, we think the useful question is not "can we build this?" Most strong teams can. The question is what it costs to keep it correct, affordable and trusted a year from now.
A working prototype proves the model can do the task. It says very little about the harness: the routing, context, guardrails, evaluation and plumbing around the model that make it safe to run unattended across every repository.
That harness is where most of the cost lives, and almost none of it is visible in the demo.
In February 2025 we shipped a screening stage at the front of our triage pipeline: one structured call to Claude 3.5 Sonnet v2, roughly 40 lines of code around the finding, and no ability to look further into the repository. It was a sound design for the models of the time, and it caught about 88% of eventual false positives before they reached the expensive stages.
By mid-2026 the assumptions underneath it had changed. Open-weight models could run reliable multi-turn tool loops at roughly a tenth of frontier token prices. We rebuilt the stage as a cheaper model that could Read, Grep and Glob the repository itself. In a controlled comparison it used 97.5% fewer input tokens, ran in 18.4 seconds instead of 592.9, and got the answer right where the old design had deferred it.
The lesson for a build decision is not about that one stage. A harness is never finished. The best architecture for a given model generation becomes the expensive one within eighteen months, and someone on your team has to notice, redesign, re-evaluate and migrate it.
Your harness depends on scanners, source control, ticketing, chat and cloud APIs. Each has its own data model, severity scale and release cycle. Every one of them is an integration you now own.
When a scanner changes its output format or a model provider deprecates an endpoint, nothing warns you. Findings quietly stop flowing, or start arriving with the wrong severity. The fix is usually small. Finding it is not, because the person who wrote the wrapper is now working on something else.
Across five production tenants, we looked at 14,847 SAST findings triaged over 90 days. 57% turned out to be false positives. Full-depth investigation of a single finding costs roughly $0.40 to $1.60 at frontier rates.
A harness that sends every finding down the deepest path spends most of its budget carefully confirming noise. Avoiding that means deciding which questions can be answered by a cheap model with the right evidence, and which need expensive reasoning. That routing logic is not a setting. It is ongoing engineering work, and it has to be revisited every time prices or models change.
An AI system that can dismiss findings or open pull requests needs a failure contract. In our pipeline, the screening agent may only dismiss a finding when it can point to affirmative evidence that it is benign. If it is uncertain, runs out of budget or hits a tool error, the finding goes to full triage unchanged. A wrong deferral costs some compute. A wrong dismissal hides a real vulnerability.
Designing that asymmetry is only the start. You also need an evaluation set that reflects your code, a way to detect when a model upgrade makes things worse, and a process for checking that fixes actually close the vulnerability without breaking anything. None of that ships in a weekend, and none of it can be switched off once developers depend on the output.
Add these up and the largest cost is not tokens or licences. It is the engineers who quietly become the owners of an internal AI platform.
They field the on-call page when triage stalls. They debug why one repository produces nonsense fixes. They answer the auditor who asks how an automated dismissal was justified. Every hour they spend there is an hour not spent on threat modelling, design review or the product itself.
The prototype costs a weekend. The harness costs a team.
Before committing to build, write down who will own each of these for the next twelve months:
If every line has a named owner with the time to do it, building may be the right call. If most of them say "the security team, in spare time", the true cost is already higher than the plan assumes.
Nullify provides the orchestration layer out of the box, so your engineers can focus on security outcomes rather than AI plumbing.
Routing and context, maintained for you. Triage agents investigate findings from SAST, SCA, secrets and cloud posture with the repository and asset graph in reach. Cheap models investigate where evidence settles the question, and deeper reasoning is reserved for the findings that need it.
Guardrails built in. Findings are only dismissed with evidence. Exploit validation proves which issues are real before anyone is asked to act on them.
Output that fits existing workflows. Validated findings become merge-ready fix PRs in GitHub, GitLab or Buildkite, and your developers review and merge them as usual.
When the next model generation changes the economics again, re-architecting the harness is our job, not yours.
A weekend prototype answers whether AI can do security work. It does not answer who will keep that system correct, affordable and trusted every week after.
Before you build the harness, price the year after the demo.
