Evaluation rubrics: defining 'good' for your AI product

An eval rubric is the written definition of "good" for an AI product: the criteria an experience should meet, given the user's intent, scored on an anchored scale.

That one document does more work than almost anything else your team writes. It's how Google has measured Search quality for two decades, how Amazon defines shopping relevance, and how the best AI teams decide whether a launch actually made things better. This is the primer: what a rubric contains, what one looks like, and how to write one that measures what your users actually care about.

What goes in an evaluation rubric

A complete rubric has four parts:

• The criteria - what matters on this surface, given what the user came to do. For marketplace search: relevance to the intent, correct size, listing quality, seller trust, no duplicates near the top.
• An anchored scale - usually 0 to 3, where every score has a written "why." "3 = highly relevant" produces arguments. Writing down why a 3 is a 3 produces agreement, whether the judge is a person or a model.
• Follow-up questions - the prompts that capture the reasons behind a score, so you learn what to fix, not just how you did.
• What the judge sees - exactly which snapshot of the experience gets judged: the rendered screen, the listings with their photos, the feed shown to this user at this moment. Not the logs.

What a rubric looks like

The image at the top of this post is a real one, compressed. The intent behind it: "I searched 'vintage Levi's jacket size M' because I want one that actually fits my style." The scale:

• 3 - Satisfies the intent outright: vintage Levi's, size M available, consistent with her style, from a trustworthy listing.
• 2 - Close substitute: right style and size but a different era, or the right jacket with a minor listing issue.
• 1 - Keyword-relevant but intent-poor: Levi's in the wrong size, vintage but the wrong garment, or a duplicate of a higher result.
• 0 - Fails the intent: irrelevant, misleading, or untrustworthy.

Ten minutes of writing, and "is search getting better?" becomes a number you can track release over release.

Not the classroom kind

A note on the word: most of what the internet calls an "AI rubric" is a grading rubric a teacher generates with AI. Same word, opposite direction. An evaluation rubric for an AI product is how you grade the AI - and the graders whose opinion decides are your users.

Which scale: 0-3, pass/fail, or side by side?

• 0-3 with anchors - the default. Enough resolution to see movement, few enough levels that judges agree.
• Pass/fail - the reward for granular intents. When the intent is specific enough, the rubric collapses to one question: did this satisfy it?
• Side by side - for close calls and launch decisions. Show the judge two versions against the same rubric and have it pick a winner. Google's raters have worked this way for decades, because comparison is more reliable than absolute scores.

The two ways rubrics go wrong

Borrowed criteria. Hallucination rates, toxicity flags, benchmark metrics. These grade your product against the research field's intent, not your users'.

Fiat criteria. The smartest person in the room writes down what good means, and nobody checks. Sharp, opinionated, unvalidated.

The fix is the same for both: treat the rubric as a hypothesis about your users, then validate it. Put real assets in front of real users with structured ratings and reasons, measure where their judgment disagrees with your rubric, and rewrite the anchors in their words. A validated rubric is the voice of the customer in executable form. It's also what makes an LLM judge trustworthy enough to run at scale, because a judge is only as good as the rubric it applies and the human ground truth it's calibrated against.

Writing your first one

• Pull twenty real sessions and write the intent behind each in one sentence.
• Draft a one-page rubric for one surface: 0-3, anchored why at every level.
• Grade fifty assets yourselves. Where teammates disagree, tighten the anchors.
• Have real users grade the same assets, with reasons. Rewrite your anchors in their words.
• Hand it to an LLM judge, and re-calibrate on a cadence, because criteria drift and so do users.

For the bigger argument - system quality versus experienced quality, intents, judges, and the loop that runs it all - read How good is your AI product?. If you'd rather start from a rubric sharpened by your own users, Grade recruits them, records them using your product, and turns what they say into a rubric and an autograder. Join the Grade waitlist.