What Is LLM Evaluation?
LLM evaluation is the practice of measuring how well a language model or AI system performs a specific task, using test cases with known answers and clear success criteria, so teams can compare models and catch regressions before changes reach users.
Also known as:
How LLM Evaluations Work
An evaluation, or eval, starts with a clear definition of success, such as the share of findings correctly classified or the share of fixes that pass tests. It then runs the system against a set of test cases and scores the results.
- Exact checks. Compare output to a known answer, such as true or false positive.
- Code-based checks. Run tests, builds or scripts against what the model produced.
- Model-based grading. Use another model, with a clear rubric, to grade open-ended output.
- Human review. Have experts grade a sample, especially for new or high-stakes tasks.
Why Evals Matter in Security
Public benchmarks say little about how a model will perform on your code, your stack and your findings. An evaluation set built from real, labeled cases tells you whether a new model, prompt or routing rule actually improves results, and whether it quietly makes something worse.
What to Measure
- Accuracy on both sides: real issues missed and false ones kept.
- Cost and latency per task, since a better answer at ten times the price may not be worth it.
- Consistency across repeated runs.
- Behavior on edge cases, such as unusual languages, very large files or tool errors.
Evaluation Is Continuous
Models are updated, deprecated and replaced often. Every change to a model, prompt, tool or routing rule should be checked against the evaluation set before it ships, the same way code changes are checked against tests.
How Nullify Approaches Evaluation
Nullify measures architecture changes against controlled comparisons before adopting them, and treats proof as the standard for output: exploits are independently re-reviewed before they are reported, and proposed fixes are checked against builds and tests before a PR opens. See how Nullify proves findings and fixes.
Key Takeaways
- LLM evaluation measures task performance against test cases and clear success criteria.
- Public benchmarks rarely predict performance on your own code.
- Measure misses, false alarms, cost and consistency, not just accuracy.
- Re-run evals before every model, prompt or routing change.
Read More About LLM Evaluation
Exploit Validation
See Nullify in Action
Nullify proves findings and fixes before they reach your team.