Nullify

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

Product Security Terms›
LLM-as-a-Judge

Product Security Terms

LLM-as-a-Judge

NT
Tim Thacker
November 14, 2026

What Is
LLM-as-a-Judge
?

LLM-as-a-judge is an evaluation method where one large language model grades the output of another model, or its own, against a rubric or a reference answer. It lets teams score open-ended output at scale when human review would be too slow or expensive.

Also known as:

LLM judge, model-graded evaluation, AI judge

How It Works

A judge model receives the task, the output to grade and instructions on what good looks like, then returns a score, a pass or fail, or a preference between two answers. Common setups include grading a single answer against a rubric, comparing two answers side by side and checking an answer against a known reference.

Where the Method Comes From

The 2023 paper "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" by Zheng et al. studied the approach in depth. It found that strong LLM judges matched human preferences more than 80% of the time, about the same rate at which humans agree with each other, and it also documented the method's biases.

Known Weaknesses

  • Position bias. Preferring whichever answer appears first or second.
  • Verbosity bias. Rating longer answers higher, even when they are not better.
  • Self-enhancement bias. Favoring answers written by the same model or model family.
  • Limited reasoning. Missing errors in math, logic or code that the judge cannot verify itself.

Why Judges Are Not Enough for Security

For security work, a judge's opinion that a vulnerability is real or a fix is correct is still an opinion. Stronger evidence comes from checks that can be run: does the exploit reproduce, does the fix build, do the tests pass, and does the original attack fail after the patch. Good evaluation pipelines use LLM judges for what cannot be executed and deterministic checks for everything that can.

How Nullify Verifies Results

Nullify does not rely on a model's opinion alone. Proposed fixes are independently reviewed and checked against builds and tests before a pull request opens, findings from the pentest agent come with request and response evidence, and developers approve every merge. See how Nullify proves findings and fixes.

Key Takeaways

  • LLM-as-a-judge uses one model to grade another model's output.
  • Research found strong judges agree with human preferences more than 80% of the time.
  • Judges show position, verbosity and self-enhancement biases.
  • In security, pair judges with executable checks such as builds, tests and exploit replays.

Read More About
LLM-as-a-Judge

Exploit Validation

See Nullify in Action

Nullify backs every finding and fix with evidence you can rerun, not a model's opinion.

Explore Exploit Validation

Reduce your vulnerability backlog and lower your MTTR

The AI harness that finds, proves and fixes vulnerabilities across your codebase.

Book a demo