Why allocating frontier model capacity is becoming as much an economics problem as an accuracy problem, and what that means for teams deciding whether to build their own AI security harness.
Whenever I speak with security teams, one of the most important questions I hear them asking right now is how they should actually deploy frontier model capacity across vulnerability discovery, triage and remediation.
That question becomes much more interesting once experimentation turns into production. It is easy to make an individual security task better by giving it the strongest model available, a huge context window and enough agent turns to investigate until it reaches a conclusion. It is much harder to do that economically across tens of thousands of findings, multiple stages of triage and remediation, and an increasingly large amount of AI work running continuously in the background.
After more than three years of building agents at Nullify to perform application security work, we have increasingly come to think about this as an economics problem as much as an accuracy problem.
Our recent work rebuilding one of the earliest stages in Nullify's triage harness is a good example. We replaced a one-shot frontier model false-positive classifier with a much cheaper open-weight model that could investigate the repository itself. The result was counterintuitive: fewer tokens consumed, lower cost, and more accurate triage.
The interesting lesson is not that cheaper models are somehow better than frontier models. It is that the harness determines where expensive intelligence is actually necessary, and as every security team becomes a manager of finite AI resources, that increasingly becomes the real question underneath the build versus buy decision.
False-positive classification sits near the beginning of Nullify's triage pipeline, which makes the economics of the stage particularly important.
Across five representative production tenants, we looked at 14,847 SAST findings triaged over a 90-day period, of which 57% were ultimately classified as false positives. Giving every one of those findings the full treatment of context gathering, exploitability analysis, severity drafting and prioritization would cost roughly $0.40 to $1.60 per finding at frontier model rates.
In other words, if you simply sent every scanner finding into the deepest possible AI investigation, a very large amount of the budget would be spent carefully proving that noise was, in fact, noise.
So in February 2025, we introduced a screening stage before full triage. The design reflected the model landscape at the time: Claude 3.5 Sonnet v2, one structured model call, roughly 40 lines of code surrounding the finding, and no ability to inspect additional repository context. The model either found enough evidence to dismiss the issue or deferred it into the rest of the triage pipeline.
At the time, this was a very rational architecture. Frontier models were the only models we trusted for this kind of security judgment, open-weight models were materially weaker at instruction following and structured output, and reliable agentic tool use below the frontier was still immature. Token prices also made "cheap" relative, so the practical way to make the screen economical was to constrain the amount of context we gave it.
And the system worked well. Across the 90-day production dataset, the screen caught roughly 88% of eventual false positives before those findings reached the expensive downstream stages.
If you were building an internal AI triage harness in early 2025, this is close to the architecture you might reasonably have shipped yourself. Take scanner output, give the model some local code, call the best model you can afford, keep the context bounded and send the uncertain cases into a deeper path.
The important part is what happens eighteen months later, when the assumptions underneath that architecture have changed.
Across the same dataset, 1,025 findings were deferred by the screen and later classified as false positives anyway, representing roughly 12% of all eventual false positives.
The obvious response might be to treat those misses as a model quality problem. Upgrade Sonnet, move to Opus, increase the reasoning budget or give the model more tokens.
But when we looked at the findings that were leaking through the screen, the problem was often much simpler. The model did not have access to the one fact it needed to make the decision.
A hash function might look like a broken security control if you inspect the function itself, but its caller reveals that it is only being used to generate a cache key. A raw SQL fragment might look injectable in the local window, but following the operand into another file reveals that it can only ever contain a hardcoded alias.
In both cases, giving the same model more reasoning does not solve the problem because the decisive evidence is outside the context it has been given. A stronger model looking at the wrong evidence still has the wrong evidence, while one Grep into the repository may answer the question immediately.
This was the architectural ceiling of the original design, and it is also where the economics of a roll-your-own harness start becoming much less obvious.
By mid-2026, the constraints that had shaped the 2025 architecture had changed materially. Open-weight models could run reliable multi-turn tool loops, hosted inference had become available at roughly one-tenth of frontier per-token prices, and the capability gap had narrowed enough that for a tightly bounded question answerable from concrete evidence, access to that evidence could matter more than additional model scale.
So instead of replacing the old screen with an even more capable frontier model, we changed the architecture around the model.
The new false-positive screen starts with the exact same seed context as the old one, but instead of asking a frontier model to make a one-shot decision, we run a cheaper open-weight model as a bounded investigation agent.
The agent gets three read-only tools over the repository at the exact scanned commit:
Read. Grep. Glob.
That means it can follow a symbol into another file, inspect the caller, trace where a value came from, inspect configuration or schema, and gather the particular piece of evidence it needs before making a decision.
The most important part of the design, however, is not just tool access. It is the authority we give the agent.
The screen is only allowed to dismiss a finding when it can point to affirmative evidence that the finding is benign. If it is uncertain, exhausts its budget, encounters a tool failure, hits an infrastructure error or simply cannot establish the answer, the finding falls through into the existing full triage pipeline unchanged.
That gives us a deliberately asymmetric failure model. A wrong deferral costs us some additional downstream compute. A wrong dismissal is the dangerous failure mode, so dismissal gets a much higher evidence bar.
This distinction matters when you think about building AI security systems internally. We did not simply ask whether a cheaper model could imitate the frontier model. We changed the task, the context, the available tools, the stopping condition and the failure contract until the cheaper model had a job it could perform safely and economically.
That is the job of the harness.
Before enabling the new screen on customer traffic, we ran both architectures end to end on a controlled pair of findings with the same adversarial shape. In both cases, the scanner had flagged a weak-hash issue whose benign purpose could only be established by following the relevant code into its caller.
The difference was how each architecture was allowed to gather context.
| One-shot frontier screen | Tool-using open-weight agent | |
|---|---|---|
| Input tokens | 187,862 | 4,718 |
| Wall-clock | 592.9s | 18.4s |
| Cost at frontier rates | $0.2122 | $0.0218 |
| Cost at open-weight rates | — | $0.0026 |
| Outcome | Deferred | False positive |
The agent used 97.5% fewer input tokens.
More importantly, it resolved the finding correctly. The one-shot path could not see the evidence it needed, so the finding spilled into full classification. The agent used Read and Grep to follow the relevant symbol, established that the function was being used purely as a content checksum rather than as a security control, and stopped.
The point is not simply that 4,718 is a smaller number than 187,862. The point is that the architecture retrieved the evidence required for the decision instead of carrying substantially more context through a deeper and more expensive pipeline.
That gave us the mechanism. Production told us whether the economics and accuracy would hold across real findings.
During the first nine days of production, the new agent screened 682 findings across six tenants and dismissed 445. The average investigation took 3.9 turns and cost $0.0103 per call.
At the same time, tenants that had not yet migrated were still running the previous one-shot screen, which gave us a useful contemporaneous cost comparison.
| Architecture | Model | Avg. turns | Avg. cost per call |
|---|---|---|---|
| Tool-using agent | Qwen 3.7 Plus | 3.9 | $0.0103 |
| One-shot screen | Claude Sonnet | 1.5 | $0.0305 |
The new agent performed roughly 2.5 times as many turns and still cost only one third as much per finding.
This is why we think simply counting model calls or trying to minimize turns is the wrong way to think about token economics.
A system can perform more turns and still be dramatically cheaper if the work is being done at the right model tier, with tightly bounded context and a clear stopping condition. What matters is the cost of producing the final security outcome, not how few times the model was called along the way.
The more important question, though, was whether we had traded accuracy for that lower cost.
We had not.
Against the findings database as ground truth, the new screen caught 99.1% of eventual false positives before the full classifier ran.
Contemporaneous control tenants running the one-shot frontier screen on the same classifier path caught 94.9%.
That means the proportion of false positives that escaped screening and unnecessarily paid for full analysis fell from 5.08% to 0.90%, which is a 5.6 times reduction in miss rate.
This is the part of the result we find most interesting because it runs against the way most teams naturally think about model allocation.
Normally, if you push work from a frontier model onto a cheaper model, you expect some quality degradation and then decide whether the savings are worth it. Here, we got the opposite result. The cheaper model produced a better outcome because the architecture gave it access to evidence the one-shot frontier model could not inspect.
In this case, lower cost and higher accuracy were not competing objectives.
The bottleneck was not simply intelligence. It was how intelligence was being deployed.
The manual audits make this much more concrete.
We inspected dismissals against the repositories at the exact commits that had been scanned. The successful investigations included cases where a user-looking value was actually traced to a constant, where a supposed SSRF input had no attacker-controlled path, where an apparent SQL injection received only a hardcoded alias, where an apparently unsafe JWT flow was guarded elsewhere, and where a supposedly dangerous execution path existed only inside a manually invoked CLI or build context.
In the stratified audit, 55% of dismissals required opening a file other than the file originally flagged.
That is exactly the kind of evidence a one-shot local-window classifier cannot inspect, regardless of how capable the underlying model is.
Across the 46 dismissals we manually audited, zero real vulnerabilities were dismissed. Two cases contained incorrect supporting evidence because a pre-existing checkout defect had given the agent an incomplete repository clone. Manual inspection still showed that both findings were false positives, but the incident exposed an important point about production AI security systems: the infrastructure around the model becomes part of the security boundary as well.
Once an agent has real authority inside the security program, you are no longer just maintaining a prompt. You are maintaining retrieval, repositories, tool permissions, budgets, failure behavior, auditability, evaluations and the systems that determine what the model is allowed to conclude.
A phrase we hear increasingly often is some variation of, "We can just build this ourselves with Claude."
And in many cases, that is true.
The models are becoming capable enough that reproducing an individual AI security capability is increasingly straightforward. A security engineer can take scanner findings, provide some repository context and get a useful first version of triage working quickly. The same is increasingly true for remediation, code review, exploit investigation and other security workflows.
But that is not really the hard part anymore.
The hard part is continuously determining which model should perform which task, how much context it should receive, whether that context should be supplied up front or retrieved dynamically, which tools it should have access to, how many turns it should get, what evidence is sufficient for autonomous action, what happens when the underlying infrastructure fails, and when work should escalate to a more expensive reasoning path.
Then the models change.
Pricing changes.
Inference providers change.
Tool-use reliability changes.
Your own production traffic teaches you that an architecture which looked correct twelve months ago has an avoidable cost ceiling you could not see when you originally built it.
Our 2025 false-positive screen was not badly designed. It reflected the frontier model landscape of 2025. The problem is that by 2026, that landscape had moved far enough that the better architecture had become almost the inverse of the original one: a cheaper model, more turns, more tool use and less total cost.
This is the part of the build-versus-buy calculation that is easy to underprice. The comparison is not simply a vendor licence versus an API bill. If you choose to own the harness, you also choose to own the continuous optimization loop around the models.
False-positive screening is only one small stage of Nullify's broader triage system, but because it sits in front of several expensive downstream stages, its unit economics are unusually easy to measure.
The agent costs $0.0103 per finding. When it successfully dismisses a finding, it avoids downstream work costing roughly $0.55 to $0.76.
At those numbers, the screen only needs to dismiss around 1.4% to 1.9% of findings to pay for itself. In production, it was dismissing 65%, which means the deployment was operating roughly 35 to 48 times above its break-even rate.
During the initial deployment window, $7.18 of screening spend displaced $273.90 of downstream classifier spend, or roughly $38 of avoided downstream work for every dollar spent on screening.
This is the kind of accounting security teams are going to have to perform more often as AI becomes part of the operating model of the security program.
The question is not simply how much a model costs per million tokens. It is what security outcome those tokens produce, what additional work they prevent, and whether that work should have been routed to that model in the first place.
A cheap model call that creates another artifact for a human engineer to manually investigate can still be expensive. A more involved agent run that conclusively removes a finding from the pipeline and prevents several downstream stages from running can be extremely cheap.
The unit that matters is increasingly security outcome per token spent.
The wrong takeaway from this research would be that security teams should stop using frontier models and push everything onto open-weight alternatives.
That is not what the data says.
There is still a large amount of security work where deeper reasoning is justified. In our historical dataset, around 83% of triage spend was legitimate deep analysis of real findings, and this experiment deliberately did not attempt to remove that work.
There will continue to be vulnerability investigations, exploit validation tasks and remediation problems where the best frontier model available is exactly where you want to spend your budget.
The objective is to make sure those are the cases receiving it.
That means matching the model tier to the task, giving the model the right context and tools, defining the problem tightly enough that the agent knows when it has enough evidence to stop, and escalating only the work that genuinely requires more expensive intelligence.
This is why we increasingly think the important question in AI security is not "which model should we use?"
It is how should we allocate finite AI capacity across the security program to produce the greatest number of security outcomes?
Every security team is becoming a manager of finite AI resources, and the teams that get the most leverage from those resources will not necessarily be the teams deploying the strongest model everywhere. They will be the teams with a harness that can continuously determine which work deserves frontier capacity and which work can be pushed down to cheaper models without sacrificing the outcome.
For false-positive triage, our answer turned out to be surprisingly simple:
let the cheap model look.
