Users as your golden dataset: how to build evals before launch

Most eval advice builds a golden dataset from production traces. Pre-launch there are none, so teams fill it with PM guesses, simulated users or a few people. A golden dataset is the fixed set of scenarios, each paired with what good looks like, that every prompt or model change gets scored against. I think the usual ways of filling it before launch are the wrong place to start. Your real users should be your golden dataset, hundreds of them, built into how the product gets made.

Every early answer runs out of people

The best eval advice for product teams comes from Hamel Husain and Shreya Shankar, and I agree with nearly all of it. Their method starts from around 100 production traces, read and annotated, with "a single principal domain expert" as the arbiter of quality. Every criterion traces back to something a user did.

That breaks down at the first launch. Before it, there's nothing to read, and every answer I've seen for going earlier falls back on one of three things.

The first is the all-knowing PM. Someone on the team still decides what good means, but with no traces, the only evidence is the PRD. The eval set becomes a list of what the team believes users want.

The second is simulated users. Husain and Shankar suggest generating synthetic traces by having an LLM simulate user queries, and they're candid that "sometimes it's better than nothing." Someone still has to tell the model who to simulate, and that's the team again.

The third is a handful of people: a few design partners, a dozen prototype sessions, a one-off study. These are real users, and they'll teach you things. But a dozen people can't tell you how often a failure happens, which segments it hits, or what the long tail of requests looks like. Husain and Shankar's rule of thumb is to annotate around 100 traces yourself so you don't stop looking for failures too early. Before launch, few teams have 100 traces, and fewer have 100 from different people. A dozen people also can't give you enough judgments to calibrate a judge.

On Reddit, someone prepping a launch with zero users wrote, "I'm manufacturing the ground truth out of thin air." When teams calibrate a judge, one person usually labels the first batch, a commenter pointed out, "and the judge ends up calibrated to one person's intuitions."

Even the synthetic-data advice comes with a caveat: "We recommend getting real users instead of relying on synthetic data where possible." I'd take that further. You can get real users before launch, in the hundreds, if you build for it.

Serious quality programs don't run on one person's taste

The quality programs I admire most all made human judgment a standing system, with written criteria and a lot of people behind them.

Google Search's rater guidelines now run 182 pages. When I was at Google, we launched thousands of improvements to Search a year on the back of that rating loop. Amazon's ESCI dataset holds 2.6 million manually labeled judgments of whether a product is an Exact match, a Substitute, a Complement or Irrelevant for a shopping query. Harvey's Legal Agent Benchmark scores more than 1,200 agent tasks against over 75,000 expert-written rubric criteria, and Applied Compute has already used those task rubrics to train a legal agent with reinforcement learning.

Look at who the humans are, though: trained raters, annotators and domain experts. For search relevance or legal correctness, those are the right people. For a product experience, the experts are your users.

Fit belongs to your users

DoorDash's design team drew this line well in September. Nam Pham and Andrea Cavicchi sort eval questions by who can answer them. Experts or a community set the bar on Authority questions. Only the individual can judge Fit, like whether a recommendation is right for this person tonight. Most of what makes a product experience good is Fit.

Their case study shows what happens when a team writes Fit criteria on users' behalf. The reasons they'd listed for rejecting a personalized homepage carousel "had never been tested with consumers." When users explained their rejections in their own words, people who had picked "cuisine mismatch" almost never mentioned cuisine, and reasons like "I could just make this at home" and "it doesn't travel well" weren't on the list at all. Most judges trained on the corrected labels improved recall on a held-out set.

DoorDash ran this on a live feature, so it isn't a pre-launch story. It shows what a company with its own eval research group can do as a study. Most teams shipping an AI feature don't have that group.

Build it into the product

I think every AI feature should be built this way. Recruit hundreds of people who match your target user and put the prototype in front of them on tasks from their own lives. Record what they do and say, and have them rate what they got and explain why.

The scenarios then come from what people came to do, weighted by how often each kind of request shows up. The rubric's anchors use their language instead of the team's. There are enough judgments to calibrate an autograder and test it on examples it hasn't seen. When users split on an output, you can see whether the split follows a segment. With a dozen sessions, you'd be guessing whether it's noise.

Take a made-up case. Your PRD for an AI trip planner says a good itinerary fills every day. A dozen sessions might show you a few people deleting afternoon stops. Hundreds will tell you whether that's everyone or mainly families, and give you enough ratings to teach a judge the difference.

I split quality in two: system quality, graded against what the builder intended, and experienced quality, graded against what the user intended. A PRD can only describe the first. Teresa Torres has called evals "a new discovery habit", and for a team that already talks to users, this is that habit with hundreds of people instead of a handful.

Once the autograder agrees with your users, it can score hundreds or thousands of scenarios before launch, and again on every prompt or model change. You'll still do error analysis on production traces after launch. It goes faster when the rubric you're revising came from your users in the first place.

We built Grade to make hundreds of real users a standard part of building an AI feature. We recruit your target users and record them using your product, help you build a rubric from what they say and do, and deliver an autograder calibrated to them, which recalibrates once you're live. Join the Grade waitlist.