LLM as a judge vs human evaluation

Most comparisons of LLM as a judge vs human evaluation set them up as rivals. A judge only scales human judgment, though, so the humans you pick set its ceiling.

Most guides agree on the practical answer. Start with automated checks, calibrate an LLM judge against human labels, then let the judge run at scale. That advice is right as far as it goes. It rarely asks whose judgment the judge should match, and when it does, the default answer is subject-matter experts. That choice decides whether your evals measure your product or your team's opinion of it.

What each one does well

An LLM judge is a language model prompted with your rubric and asked to grade. It's fast and cheap next to people, and it never gets tired. Strong LLM judges agreed with expert and crowdsourced human preferences more than 80% of the time on MT-Bench and Chatbot Arena, the same rate at which the humans agreed with each other. Anthropic tells teams to automate grading wherever they can and favor volume over a smaller set of hand-graded examples.

LLM judges also fail in known ways. They tend to prefer longer answers and whichever answer appears first. They can also be wrong with total confidence. When OpenRouter padded 36 wrong answers with two true sentences each, an LLM judge flipped three of them from rejected to accepted, and every flip went from a confident no to a confident yes.

Human evaluation makes the opposite trade. People notice when a technically correct answer misses the point, understand what someone meant, and can explain in their own words why something failed. They're also slow and expensive per judgment. They disagree with each other, and few teams can put people on every release.

Each covers the other's weak spots, so most guides recommend both. Teams differ on who the humans are.

Every judge is calibrated against somebody

In September, TypeSafe AI launched Jev, a decision model that several eval tools now support as a judge. Within two weeks, three benchmarks had tested it against other judges, and each said plainly what it used as the reference.

• LangChain measured accuracy against one human reviewer, who labeled five captured runs of a weather agent.
• OpenRouter used 88 items from a public faithfulness dataset, which the coding agent running the experiment had reviewed, pruned and partly relabeled.
• A paper by Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman used published benchmark labels, with a researcher adjudicating disagreements blind. That adjudication found 24 of the 26 HaluEval items both judges "missed" carried labels the evidence didn't support.

Each reference fits what its benchmark tests, which is whether a judge can grade a well-defined question correctly. None of them set out to measure agreement with a product's users, and nobody should expect them to. That 80% figure has the same scope. It measured agreement with experts and crowdsourced voters judging chat answers.

Product teams have their own usual references, and each one knows something different.

Your team knows what the system was built to do. When engineers and PMs label the examples, you calibrate the judge to the builder's intent. That's system quality, and it's worth measuring.

Hired annotators and experts know the guideline you gave them, or the domain. They're the right reference when correctness needs credentials, like a legal or medical answer. They aren't your customers.

Your users know what they came to do. They're the only reference for experienced quality, meaning whether a real person got what they came for and how it felt along the way. Before launch, that means recruited target users, and Hamel Husain and Shreya Shankar recommend real users over synthetic data wherever you can get them.

A judge calibrated to the wrong group produces clean, consistent numbers that point your roadmap somewhere your users aren't. LangChain warns that a consistently wrong evaluator can produce bad feedback at scale.

Two meanings of calibrated

Most of September's Jev coverage called it "calibrated," and the word means two different things.

The first meaning is statistical. A judge is calibrated when its confidence means something. Of all the verdicts it gives a 0.9, about 90% turn out right. TypeSafe trains Jev specifically for this. Jev's Brier score, which measures the gap between stated confidence and actual outcomes (lower is better), was 0.043 against 0.054 for GPT-5.6 Luna when OpenRouter tested both. Across 88 cases, Luna's confidence took only six distinct values. Statistical calibration is what lets you set a threshold and route uncertain cases to a person.

The second meaning is agreement. A judge is calibrated to your product when its verdicts match the judgment of the people whose opinion defines good for your product, measured on the same experiences. When product teams talk about LLM judge calibration, this is the one that decides whether the numbers mean anything. It's the step in AI product quality work where real users grade the same experiences the judge grades, and you measure how often they agree.

A judge can be well calibrated in the first sense and badly calibrated in the second. Then it tells you, with well-founded confidence, how closely your product matches a standard your users don't hold. The model vendor trains for statistical calibration, though you still check it on your own labels. Agreement calibration is yours alone, and no model release will do it for you.

Without a reference, no judge can grade

Li's team gave Jev and two GPT models natural prose to judge with no reference answer and no source document. All three scored near chance, between 53.5% and 56%, while reporting average confidence of 91% or higher. None of the judges they tested could grade prose reliably without a reference.

A faithfulness check has its reference built in, the source document the answer should stick to. Most of what users care about has nothing like that. Whether a page of search results fits a shopper's style, or whether an onboarding flow leaves someone confident enough to keep going, has no source passage to check against. The reference has to be built, and the users whose intent defines the answer are the only people who can build it.

In practice that reference is a golden set. It's a collection of treatments, meaning snapshots of what users saw and got, each graded by real users on your rubric's scale with the reason behind every grade. How far you can trust your judge depends on how often it agrees with that set.

How to calibrate an LLM judge against real users

The method is the same for an LLM judge, Jev, or a cascade of both.

1. Write the intents and a rubric for each surface. An intent is the user's goal in one sentence, with enough context to think like them. A rubric is the definition of good for a treatment, given an intent, with an anchored reason for every score.
2. Sample treatments that look like reality. Pull 50 to 100 per criterion, the range OpenRouter recommends for setting thresholds, weighted toward what your users actually try to do. If you haven't launched, that mix comes from recruited target users trying the product. Include the hard cases: close calls, polished wrong answers, rare intents, and the same pair shown in both orders.
3. Recruit users who match your customers. Show them the same treatments and collect a rating on the rubric's scale, plus the reason for it. Screen for the segments that matter. Your team, your annotators, your most active users and a single principal domain expert can each skew the set in their own direction.
4. Run the judge on the same treatments and measure agreement. For pass/fail criteria, check how often the judge passes what users passed and fails what users failed, because raw percent agreement looks great when most cases pass. Use a chance-corrected statistic like Cohen's kappa for categories, and rank correlation for ordered scales. Then check statistical calibration by grouping the judge's verdicts by confidence and seeing whether its 0.9s are right about 90% of the time.
5. Read every disagreement. Where the judge and your users differ on experienced quality, go with your users unless their reason shows a mislabel. Rewrite the rubric's anchors in their words and rerun. When DoorDash's design team tested its rejection reasons for homepage carousels with consumers, people gave reasons the team's list didn't include, like "I could just make this at home." Where users disagree with each other, the intent is probably too broad, so split it.
6. Set thresholds and route. Accept the judge's confident verdicts, send the uncertain middle to a stronger judge or a person, and write down the error rate you've agreed to live with.
7. Recalibrate when anything moves. Rerun the agreement check when the rubric, the judge model or your user base changes. Model versions change quietly, and OpenRouter suggests rerunning whenever the model version does.

Humans once, judges always

Google Search keeps human raters in the loop at a scale few teams can match. External raters work from 182 pages of guidelines, and proposed changes to Search are rated side by side against the current results before launch. Google ran 124,942 of those side-by-side experiments in 2023. Those raters apply a standard Google wrote, and most teams can't staff anything like that on every release. Our shorthand for the version built on your users is humans once, judges always. Real users establish what good means, and you refresh that on a cadence. The judge applies the standard to every release at machine speed, and you recheck its agreement with the users whenever something shifts.

Grade does the user side of this work for you, before launch or after. We recruit your target users and record them using your product across hundreds of scenarios, help you build a rubric from what they say and do, and deliver an autograder calibrated to them. Once you're live, Grade recalibrates it. Join the Grade waitlist.