What is LLM-as-a-judge?

LLM-as-a-judge is an evaluation method in which a language model scores or compares another model's outputs, following a rubric you write. It makes it possible to evaluate open-ended outputs, such as summaries, answers and conversations, at a scale human review cannot match, provided the judge is checked against human judgement.

3 min read·Checked September 25, 2026·Also called LLM judge, model-graded evaluation, AI grader

Read the free chapter →Prompt Engineering guide

An LLM judge is only as trustworthy as its agreement with a careful human. Used that way, it turns quality checks that would take a person days into a test that runs in minutes.

How does LLM-as-a-judge work?

You give a judge model the input, the output to grade and a rubric, and ask for a score or verdict, ideally with a short justification. There are three common setups:

Setup	What the judge sees	Output	Good for
Single-output grading	One response and a rubric	Pass or fail, or a score per criterion	Tracking quality over time
Pairwise comparison	Two responses to the same input	Which is better, or a tie	Comparing prompts or models
Reference-based grading	A response and a known-good answer or source documents	Whether it matches or is supported	Correctness and faithfulness in RAG

Does LLM-as-a-judge agree with humans?

Strong judges can agree with humans about as often as humans agree with each other, on the right kind of task. The 2023 paper that popularised the term, by Zheng and colleagues, found that GPT-4 as a judge matched human preferences over 80% of the time on their chat benchmark, similar to the agreement between human raters. That result is for general preference judgements; agreement on your specific criteria has to be measured on your data.

What biases do LLM judges have?

The same research, and much since, documented consistent biases:

Position bias: Preferring the first (or second) answer in a pairwise comparison regardless of quality
Verbosity bias: Rating longer, more detailed-looking answers higher even when they are not better
Self-preference: Rating outputs from the same model family more favourably
Leniency: Giving high scores to anything plausible, so scores bunch at the top
Surface cues: Being swayed by confident tone, formatting or keywords from the rubric

Countermeasures are straightforward: run pairwise comparisons in both orders and keep only consistent verdicts, use binary pass or fail criteria instead of 1 to 10 scales, ask for specific evidence before the verdict, and where possible use a judge from a different model family than the system being graded.

How do you write a good judge prompt?

1
One criterion at a time
Judge "is every claim supported by the sources?" separately from "is the tone appropriate?".
2
Binary where possible
"Pass if the answer states the refund window from the policy, fail otherwise" is more reliable than "rate accuracy 1 to 10".
3
Show examples
Include a passing and a failing example with explanations.
4
Reason, then decide
Ask for a short justification citing the output before the verdict, or use a reasoning model.
5
Structured result
Return JSON such as {"reason": "...", "pass": true} so scores can be aggregated; see structured output.

How do you calibrate an LLM judge?

Calibrate a judge the way you would train a new human reviewer: compare its verdicts with expert labels and fix the disagreements.

1
Label a sample
Have a domain expert grade 50 to 100 outputs against the criterion.
2
Run the judge
Grade the same outputs with the judge prompt.
3
Measure agreement
Check how often they agree, and look separately at false passes and false fails.
4
Fix the prompt
Read each disagreement; clarify the rubric or add examples.
5
Re-check periodically
Re-measure when you change the judge model or the system being graded.

Do: Measure the judge against human labels before trusting its numbers
Do: Look at its reasons, not just its scores, when results look surprising
Do: Use code checks instead whenever a criterion can be checked exactly
Avoid: Use one vague "overall quality" score
Avoid: Let the model being evaluated also be its only judge, without checking
Avoid: Treat small score changes as meaningful without enough test cases

Frequently asked questions

What is LLM-as-a-judge used for?

It is used to evaluate open-ended outputs that code cannot check, such as whether an answer is helpful, faithful to its sources, on-topic, safe or written in the right tone, across hundreds or thousands of test cases.

Is LLM-as-a-judge reliable?

It can be, for well-defined criteria and after calibration against human labels. Without calibration, known biases such as position and verbosity bias can make its scores misleading.

Should I use a 1 to 10 scale or pass/fail?

Pass or fail per specific criterion is usually more reliable and easier to act on. Numeric scales tend to bunch together and are interpreted inconsistently by both models and people.

Can the same model be the judge and the system under test?

It can work, but it risks self-preference bias. Using a different, strong model as the judge, and validating it against human labels, is safer.

Last checked for accuracy on September 25, 2026. Written by the solidcoder team.