Evals for product managers, before and after launch

Evals for product managers start with deciding what good looks like for your users. Here's what PMs own, what engineers own, and how to start before launch.

Most guides to AI evals for product managers agree on the division of labor. The PM decides what good means and turns it into criteria. Engineering builds the tooling that scores the product against those criteria on every change. When the guides get specific about data, most start from production traces, meaning around a hundred real conversations or sessions, read closely and sorted into failure modes. That's the right place to start if your AI feature is live. If you're about to ship a new one, you have no traces, and the usual substitutes are the PM's own judgment, simulated users, or a handful of real ones.

What an eval is, in product terms

An eval is a repeatable test that scores your AI product against a written definition of good. It has three parts.

• Scenarios. The situations you test: a user, their goal, and what they ask or do. We write the goal as an intent, the user's underlying goal in one sentence with enough context to think like them.
• A rubric. The definition of good for a given intent, on a scale with a written reason for every score.
• A grader. Whatever applies the rubric. That can be code, an LLM judge or people, and many teams end up using all three. Types of evals sorts them by who does the judging.

Running the grader across all your scenarios is the eval. It gives you a score you can track from release to release, or use to compare two versions before you ship either.

For a PM, an eval works like a spec that runs. A PRD says what the feature should do. An eval checks whether it did, across hundreds of cases, every time someone changes a prompt or swaps a model.

Most evals grade an output against what the builder intended. Others grade the experience against what the user intended, which is closer to whether people come back. Our guide to AI product quality calls these system quality and experienced quality. Engineering can own the first. The PM has to own the second.

What the PM owns and what engineering owns

Most PM guides describe eval ownership as shared. In practice the split usually looks like this.

Error analysis, which means reading real outputs and naming what went wrong, is shared. Engineers are faster at spotting broken behavior. PMs are better placed to say which broken behavior a user would notice.

Aman Khan describes the PM's core task as turning user expectations into precise criteria for the judge. That job description assumes you already know what your users expect, which gets harder the newer the feature is.

The standard playbook starts from real traces

The eval method Hamel Husain and Shreya Shankar teach starts with data. Pull around 100 representative user interactions. Have a domain expert read each one and write down, in plain words, what went wrong. Group those notes into failure modes. Only then decide what to measure, and automate checks for the failures that matter. Their newest work keeps the same bar of about 100 annotated traces before you automate anything.

Teresa Torres followed the same order when she built evals for Product Talk's Interview Coach. She started from real transcripts her students had submitted, and she argues that you can't know what to measure until you've done the error analysis.

Both are right that real data beats imagined data, and that the criteria worth measuring come from reading real interactions. Husain and Shankar also describe criteria drift, where reviewing examples changes your definition of good. Before launch, though, there are no interactions to read.

Before launch, the usual answers fall back on three things

Ask how to get evals right before there's production data, and the answers come in three kinds.

The all-knowing PM. Husain and Shankar recommend one principal domain expert as the final word on quality, and on a product team that's often the PM. After launch that works well, because the expert is reading what real users did. Before launch, the same expert is grading against users they haven't watched use the feature. Tezan Sahu's pre-launch playbook starts here too, with the PM and a subject-matter expert, or sometimes a customer, hand-writing 30 to 50 seed examples. It's the most thorough pre-launch method we've read, and until design partners arrive, nearly every judgment of what's good comes from a few people.

Synthetic users. Husain and Shankar describe generating synthetic traces by having an LLM simulate user queries, and they're candid that this is a fallback that is sometimes better than nothing. As they put it, "We recommend getting real users instead of relying on synthetic data where possible." Simulated users can widen the range of inputs you test, but they can't tell you what a real person counts as good.

A handful of users. Design partners (where real production traces first enter Sahu's playbook), an early-access cohort, or a one-off study with a dozen people. These are real people, and far better than none. A handful still can't give you what an eval depends on:

• Frequency. Which goals people arrive with and how often, so your scenarios are weighted the way real use is.
• Coverage. Whether new users, power users and each segment you serve mean the same thing by good.
• Volume. Enough graded examples to calibrate a judge. Husain and Shankar ask for about 100 annotated traces before you automate, and a few sessions won't get you there.

DoorDash's design team found out what their own judgment had missed by asking customers. Their first question is "who should the answer key reflect?" The reasons their team assumed customers would pass on a personalized recommendation "had never been tested with consumers." When they did ask, customers gave reasons the list didn't have, including "I could just make this at home" and food that "doesn't travel well." That was a live feature, studied by a company with its own research team. A PM shipping something new usually has neither the traffic nor the team.

Build the definition of good with hundreds of real users

Before launch, the PM's job is to put a system in place where hundreds of your target users write the scenarios and define good.

In practice, you recruit people who match your customers, across the segments you serve, and each of them brings a real goal to your prototype. You record what they do, then ask what they thought, open-ended first and then for the one main reason, which DoorDash found cleaner than a check-all-that-apply list. Then they score the result on your rubric's scale. Their goals, weighted by how often each one came up, become the scenario set. The reasons they give become the rubric's anchors, in their words, and their scores are the reference your judge is calibrated against. The recorded sessions give you a hundred-plus real traces to read, before launch instead of weeks after it.

Criteria drift still happens. It just happens before launch, when changing your mind is cheap, and the people the product is for are the ones driving it.

We think this is how AI products should be built, rather than a research step bolted onto the end. The eval set you build before launch becomes your V1 spec, as Sahu concludes, and we agree. That turns "who writes the eval set?" into "who writes the spec?" When hundreds of target users write it, you're building the product with the people it's for from the first version. Torres calls evals a discovery habit, and this is that habit at the scale an eval needs. The skills are ones PMs already use in discovery: screeners, open questions before closed ones, and people's exact words written down. The differences are the number of people and where their answers end up, in the rubric and the test set instead of a readout.

Say you're launching AI search for a resale marketplace, and one intent in your set is "I searched 'vintage Levi's jacket size M' because I want one that actually fits my style." Your team's draft rubric gives a top score to results that are vintage, Levi's, in size M and consistent with her style. That last criterion is a placeholder, because nobody on the team can say what "fits my style" means to this shopper. A few hundred people who buy secondhand clothing, each searching for something they want, can. Some might judge style by cut and wash rather than by decade. Some might fail any listing with poor photos, however relevant it is. If two segments disagree, the intent probably needs splitting. Whatever they say becomes the anchor text, and their grades are what the judge you build later has to match.

We make the longer argument for pre-launch evals in Users as your golden dataset.

A before-launch checklist

For a PM shipping a new AI feature, roughly in order:

1. Write your first intents and rubric. One sentence per intent, with the context needed to think like the user, drawn from discovery research, support tickets, and logs from any product this feature replaces or extends. Draft a 0 to 3 rubric with a written reason at every level, and treat it as a hypothesis.
2. Pilot with a few people. A small study is a good way to test your screener, your prototype and your questions before you scale up. Use it to fix the setup, not to set the bar.
3. Recruit hundreds of target users across your segments. You need enough people to see how often each goal comes up, to cover every segment you serve, and to clear the roughly 100 graded examples Husain and Shankar ask for before a judge takes over.
4. Let them bring the scenarios. Ask each person what they came to do, then record them doing it on the prototype. Their goals, weighted by how often they come up, become the scenario set. An LLM can vary the inputs to cover edge cases, but users decide which goals matter and how much.
5. Collect judgments with reasons. Open question first, then the one main reason, then a score on your rubric's scale.
6. Rewrite the rubric in their words. Where users and your draft disagree, users win. Where segments disagree with each other, split the intent.
7. Calibrate the judge against their grades. Engineering turns the rubric into code checks and a judge, and you check the judge's grades against the users' grades. LLM as a judge vs human evaluation covers how to measure that agreement.
8. Set the launch bar. Decide which intents must pass, at what score, and which failure modes block launch. Write it down before you see the final numbers.

An after-launch checklist

Launch gives you real traffic. Use it to check what your users told you.

1. Read about 100 real traces in the first weeks. Do the error analysis Husain and Shankar describe, looking for failure modes and goals your pre-launch users didn't surface.
2. Compare them with your pre-launch findings. The overlap tells you how far to trust the rubric. New goals go into the scenario set.
3. Recalibrate the judge against production. Have target users grade a sample of real outputs, re-measure the judge's agreement, and rewrite anchors wherever real use disagrees with the prototype.
4. Turn on online evals. Engineering runs the code checks and the judge on a sample of live traffic to catch regressions. You decide which score changes trigger a review.
5. Put recalibration on a schedule. Re-check the judge whenever the rubric, the model or your user base changes, and on a fixed cadence regardless.

We call this pattern humans once, judges always. People establish what good means, and judges measure it on every release. For a new feature, the human step starts before launch, with hundreds of the people you're building for, and gets refreshed on a cadence after it.

Grade runs this for you. We recruit your target users and record them using your product on the scenarios that matter, help you build a rubric from what they say and do, and deliver an autograder calibrated to them. You can grade your product on hundreds of user scenarios before a new feature ships, and Grade recalibrates the autograder once you're live. Grade is in closed beta. Join the waitlist.