How good is your AI product?

You ship a new AI feature. The evals pass. The dashboards look healthy. The launch demo got applause. So the product is good. Right?

Wrong! Or to be more precise: you have no idea. And neither does anyone else in the room, because none of those signals measure whether the product is good. They measure whether the system behaved and whether users showed up. Good is a different question.

Two different ways to measure AI product quality

AI product quality is whether your product satisfies what users actually came to do. When teams try to measure that, they end up mixing two very different things.

System quality asks: did each output pass? It lives in traces and test suites, and it's graded against what you, the builder, intended the system to do.

Experienced quality asks: did a real person get what they came for, and how did it feel along the way? It lives in the moment a user meets your product, and it can only be graded against what they intended.

These two diverge constantly, for two structural reasons.

First, quality is relational. Take a shopper who searches "vintage Levi's jacket size M." Whether the results are good depends on her taste, her budget, and what "fits my style" means to her. None of that appears in your logs. "Relevant" is not a property of the results. It's a relation between the results and a person you can't see.

Second, journeys fail by composition. Every step can pass while the whole fails. Search returns defensible results on every query, and the session is still a frustrating loop. The tutor answers correctly on every turn, and the learner still quits. Pass rates are local. Experience is global.

This is the gap between a green eval dashboard and a churn curve. It's why, when we asked 5,000 consumers about AI, the top issue they named was accuracy - as they experienced it, not as a benchmark scored it. And it's why even the analytics companies have started admitting their dashboards can't see inside AI products.

The most serious product teams write down what good means

Google Search has spent over 20 years refining its evaluation rubric. The published rater guidelines run 182 pages, and they focus on two questions: how well a result meets the user's need, and how well the page achieves its purpose.

Amazon published ESCI, a rubric for shopping queries where every result is Exact, Substitute, Complement, or Irrelevant. The label captures the relation between a result and what the shopper came for, not whether the title contains the query terms.

This is also how frontier AI teams train and improve models: put the model in an environment, score each output against expert criteria, and train it to get better scores. Harvey published a legal benchmark with 1,200+ tasks and 75,000+ expert-written rubric criteria defining what good legal work looks like. OpenAI's HealthBench does the same for health: 5,000 conversations, each with a physician-written rubric, graded by a model judge.

The common thread is simple: define what good looks like with enough specificity that you can actually measure it. That definition is called a rubric, and it's the highest-leverage document your product team isn't writing.

What a real evaluation rubric looks like

A rubric is the complete spec for judging quality: the criteria an experience should meet given a user's intent, a scale with anchored definitions of why something earns each score, the follow-up questions that capture the reasons behind a score, and exactly what the judge sees. The anchors are everything. "3 = highly relevant" produces arguments. Writing down why a 3 is a 3 produces agreement, whether the judge is a person or a model.

Here's a working example for that vintage Levi's search:

• 3 - Satisfies the intent outright: vintage Levi's, size M available, consistent with her style, from a trustworthy listing.
• 2 - Close substitute: right style and size but a different era, or the right jacket with a minor listing issue.
• 1 - Keyword-relevant but intent-poor: Levi's in the wrong size, vintage but the wrong garment, or a duplicate of a higher result.
• 0 - Fails the intent: irrelevant, misleading, or untrustworthy.

Notice what the rubric is really doing: it breaks "satisfied" into checkable pieces for one specific intent.

The same pattern holds on a completely different surface. Say your product turns rough outlines into presentation decks. The intent: "I want a polished investor pitch deck from my rough outline." The experience: the rendered slides, not the generation log. The rubric: visual hierarchy, design consistency, narrative flow from slide to slide, sensible text density, content accuracy. And because two decks can both pass while one is clearly better, the judge doesn't have to score in isolation. Show it two versions side by side - this release against the last one, or your output against a competitor's for the same outline - and have it pick a winner against the rubric. Google's raters have judged search results side by side for two decades for the same reason: on close calls, comparison is more reliable than absolute scores.

Grade against the user's intent, not yours

Most evals grade outputs against what the developer intended the system to do. Flip it. Grade the experience against what the user intended to accomplish.

An intent is the user's underlying goal, in one sentence, with enough context to think like them - the reason they showed up, not the query they typed. "I searched 'vintage Levi's jacket size M' because I want one that actually fits my style."

There's a practical reward for doing intents properly: your rubrics get simpler.

"Search quality" needs 182 pages because it's a million intents wearing one label. A single granular intent barely needs a rubric at all: did the experience satisfy it, pass or fail? The criteria stop being abstract virtues and become the components of satisfaction for that specific goal. The more granular your intents, the closer your rubric collapses toward the only question that matters. And that's universal to any product, which is why intent, not the model, is the durable center of a quality practice.

The five definitions that make AI product quality measurable

The whole system runs on five terms. Get these right and everything downstream gets easier to build, and easier to argue about.

• Intent - the user's goal in one sentence, with context. Why are you here?
• Treatment - a snapshot of what the user saw and got: a rendered screen, search results with photos, the feed shown to this user at this moment, a side-by-side of two versions.
• Rubric - the definition of good for a treatment, given an intent.
• Judge - an automated grader, usually a multimodal LLM, that scores one treatment against one rubric, given an intent.
• Eval - judges run across a whole set of intents, to score a product or compare two versions of it.

Humans drive the loop, LLMs judge it

There's an obvious objection here: if quality lives in users' heads, doesn't measuring it require asking users constantly? And users don't scale. We've heard that one before.

Right - which is why the method has two layers.

First, human ground truth, established once and refreshed on a cadence. Real people, recruited to match your customers, do three things: validate the intent set (are these the goals people actually arrive with?), anchor the rubric (their ratings and their reasons become the scale's anchor language), and calibrate the judge (grade the same experiences the judge grades, and measure agreement). Where the judge and your users disagree, your users are right, and the rubric gets rewritten in their words.

Then, calibrated judges at scale. Once the judge agrees with your users, it does what users never could: apply that judgment across hundreds of intents, on every release, at machine speed, against the actual experiences users see. An eval stops being a curated test suite and becomes a representative sample of everything your users come to do, scored.

If this sounds like how Google runs Search - sampled queries, rater guidelines, side-by-side judgment, a go/no-go for every launch - that's because it is. The method was never the secret. The infrastructure was. When I was at Google, we launched thousands of improvements to Search a year on the back of exactly this loop. What's changed is that you no longer need Google's rater program to run it.

What you get out of it

Three numbers a product team can operate on:

• Failure modes - where, specifically, your product fails intents, and why, in users' own words.
• Gold standard - what good looks like for each kind of intent. The target, made concrete.
• Headroom - the measured gap between the two. How much better this surface can get, which is what a roadmap should be prioritized by.

And the machinery pays for itself in the workflows it unlocks: launch reviews decided by experienced quality instead of vibes, pre-launch comparison of two models across simulated intents before any user sees either, regression checks that catch the degradations an error rate can't, and - because the same rubric can grade anyone's product - benchmarking your experience head-to-head against your competitor's.

How to start this week

• Pull twenty real sessions and write down the intent behind each in one sentence. Can't? That's your first finding.
• Draft a one-page rubric per surface: a 0-3 scale with an anchored why at every level.
• Grade the twenty experiences yourselves. Where teammates disagree, your anchors are too loose.
• Put the same experiences in front of real users, with structure: ratings plus reasons. Measure agreement with your own grades, and rewrite the anchors in their words.
• Only then hand the rubric to an LLM judge. And re-calibrate on a cadence, because criteria drift and so do users.

So can you actually know how good your product is?

You can't know perfectly. But you can know precisely: how good your product is for the intents you've mapped, to the degree your judge agrees with your real users, tracked release over release. That's a different sport than shipping on green dashboards and gut feel.

Google decided two decades ago that quality was too important to leave to opinion. It no longer takes a rater program and twenty years to make the same decision.

If you want help putting this into practice, book a demo. We'll show you how teams map intents, grade experiences with real users, and stand up a judge they can trust.