LLM as a judge, and whether it scores what your users score

LLM as a judge uses a language model to grade AI outputs against your own rubric. It scores what the rubric asks for, which isn't always what your users want.

In July, a team on r/AI_Agents described shipping a prompt their offline evals scored higher in almost every category, then watching their thumbs-down rate climb the same week. The new prompt wrote longer, more confident answers. Their users mostly wanted short ones that admitted uncertainty, and nothing in the eval had been told to care about that. A commenter who'd hit the same problem wrote, "The judge and the users weren't scoring the same thing."

Before I trust any LLM judge's scores, I want to know whether it's scoring what my users score.

How LLM as a judge works

A judge is a prompt that hands a model your rubric, the user's input and the output to grade, and asks for a verdict with its reasoning. The rubric does the real work. It lists the criteria and spells out what passing looks like for each one. Hamel Husain and Shreya Shankar recommend pass/fail criteria over 1-to-5 scales, because the difference between a 3 and a 4 is a matter of opinion. Some judges grade one output at a time. Others compare two versions side by side and pick the better one.

Teams use judges because nobody can read every output on every prompt change, and a judge can. Lianmin Zheng and colleagues, who introduced MT-Bench in 2023, pitched it as a scalable way to approximate human preferences, which are expensive to collect. Husain and Shankar see a judge as the only way to scale subjective evaluations that code can't check.

Zheng and colleagues found strong judges like GPT-4 agreed with human preferences over 80% of the time, the same rate at which the humans agreed with each other. They also documented position, verbosity and self-enhancement biases, meaning judges can favor whichever answer comes first, longer answers, and answers they wrote themselves. And a judge can be consistently wrong, which LangChain warns produces bad feedback at scale.

Some of those biases can be reduced. For position bias, Zheng and colleagues run each comparison twice with the answers swapped and only count a win that holds both ways. What worries me more is who the judge is agreeing with. Their agreement figures came from expert labelers and crowd voters judging chat answers. For your product, that should be your users, and a judge only knows what they want if the rubric tells it.

Is your judge scoring what your users score?

Here's the same problem in a made-up example, simple on purpose. Say you're building a cooking assistant, and a user asks it one question:

My bread didn't rise. What went wrong?

Two versions of your product answer.

Answer A

Bread usually fails to rise for one of these reasons:

• The yeast was expired or dead.
• The water was too hot and killed the yeast.
• The water was too cold to activate it.
• Too much salt or sugar slowed the yeast down.
• The kitchen was too cold.
• The dough didn't get enough time.

To fix it, check the date on your yeast, use warm water rather than hot, measure salt and sugar carefully, and let the dough rise somewhere warm until it doubles.

Answer B

The yeast is the usual suspect, but I can't tell from "didn't rise" alone. Stir a little into warm water with a pinch of sugar. If it hasn't foamed within ten minutes, the yeast is dead. If it foams, tell me whether the dough grew at all before baking and I'll narrow it down.

A rubric your team might write

Say your team writes the first rubric from its own idea of a good answer. It has three pass/fail criteria, each easy to defend in a review. The first is completeness, which the Reddit team's eval measured too.

An LLM judge applying this rubric faithfully passes A on everything and fails B on completeness. A wins, and the judge is right to pick it, because B doesn't cover the likely causes. Check the judge against your team's own pass/fail marks and it matches every one, so it passes the usual test of whether a judge can be trusted.

What users said

Now put both answers in front of people who bake at home and ask which one helped, and why. In this example they pick B, the kind of short, hedged answer the Reddit team's users wanted. Their reasons would sound something like this (my illustration, not data from a study):

• "I'm standing there with a flat loaf. Tell me the most likely thing first."
• "The first one gave me six things that could be wrong and left me to work out which."
• "I trusted the second one more because it said it couldn't tell yet."

A rubric built from what users said

Rewrite the criteria from their reasons:

Same judge, same two answers. Now B wins, and the judge agrees with your users. I'd carry the accuracy check over from the first rubric as well. Both answers pass it, so it doesn't change the result.

The judge did its job both times

Nothing about the judge changed between those two tables. It read each rubric correctly and applied it the same way. The only change was the rubric. The team wrote the first rubric from what it assumed users wanted. The second came from what users said.

You can't blame verbosity bias here either. Judges sometimes prefer longer answers for no good reason. This judge had a good reason, because the rubric asked for completeness. A perfectly unbiased judge makes the same call.

So a judge can only agree with the people its rubric came from. DoorDash's design team asked who the answer key should reflect and benchmarked their judges against an answer key from consumers. If the rubric and the answer key both come from your team's intuition, you get your team's intuition, at scale, on every release.

The obvious objection is that the team should have added a conciseness criterion. That gets you a tie at best, because on this question "complete" and "concise" pull in opposite directions. Someone still has to decide which one matters more, and the team's own rubric already chose completeness. "Concise" also isn't quite what users asked for. They wanted the likely cause first, one thing to try, and honesty about what the assistant couldn't see. A short answer that confidently names the wrong cause still fails them. You get those words by asking users. What you control is whether you hear them in a study before launch or read them in a thumbs-down rate after it.

The agreement check below also picks your judge model. Run it on each model you're considering and keep the one that agrees most often with your users.

How to calibrate an LLM judge against your users

I'll call this check calibration. Real users grade the same outputs the judge grades, and you measure how often they agree. It has two parts. Upfront calibration happens before launch, and continuous calibration keeps going once you're live.

Upfront calibration, before launch

1. Recruit people who match your target users and record them using the product on things they'd come to do anyway. Those sessions give you your scenarios, in roughly the mix your users will bring.
2. Have them grade the outputs, pass or fail, and say why. The reasons matter more than the marks, because the rubric's wording comes from them.
3. Run the judge on the same outputs and compare. For pass/fail criteria, measure the judge's true positive and true negative rates: how often it passes what users passed, and how often it fails what users failed. Keep a held-out set the judge was never tuned on for the final test.
4. When they disagree, check the rubric before you touch the judge prompt. Often the rubric measures something users don't care about or misses something they do. You can reword a judge prompt until it matches the examples you have, and it will still measure the wrong thing on the next thousand.

Husain and Shankar calibrate the judge against a single principal domain expert, working from around 100 real user interactions. That works once you have interactions to read. Before launch there aren't any, so teams fall back on the PM's judgment, synthetic users or a handful of design partners. Husain and Shankar say synthetic data is sometimes better than nothing, and recommend real users wherever you can get them. A handful of people can't tell you which complaints are common and which are one person's taste. Hundreds of real users can. I think that's how AI products should be built, with hundreds of real users writing the scenarios and defining good before launch, as a standing part of the process rather than a one-off study.

Continuous calibration, after launch

Rerun the agreement check whenever the rubric, the judge model or your users change. The commenter from that Reddit thread keeps a small human-scored sample and watches whether the judge's agreement with it drops after each prompt change. If it drops, the eval score stops meaning anything until you recalibrate. Your own standard moves too. Husain and Shankar call it criteria drift, when "reviewing examples changes your definition of good."

Once you're live, what users do becomes evidence as well. Someone who works at Conviva, an analytics vendor, described failures their eval suite never caught: conversations that scored well on helpfulness but ended with the user giving up and doing the task themselves. Their fix was to watch what users did next, like restating the request or checking the result by hand, and turn those cases into new evals.

Continuous calibration catches a judge drifting away from your users, usually after some of them have felt it. Upfront calibration points the judge at your users before any of them see the product.

Your job is still knowing what users want

None of this changes the product manager's job, which is still understanding what users want. A rubric is that understanding written down precisely enough for a model to apply. What goes in it and what to fix first are still your calls.

The judge setup is a different job. Someone has to recruit people who match your users, record them on the scenarios that matter, capture what they say and do, check the judge against them, and do it again when things change. That's the part we built Grade to handle. We recruit your target users and record them using your product. We help you build a rubric for how well their needs are met, not how good the AI sounds. Then we deliver an autograder calibrated to your users that can score hundreds of scenarios on every change, and it recalibrates once you're live.

You keep the job of knowing what users want. We make sure the judge knows it too. Join the Grade waitlist.