Synthetic users, digital twins, and what to use instead

Synthetic users promise customer research without customers. Here's where they hold up, where they fail, and why we simulate the scenario, not the person.

At IIEX West last week, nearly every conversation touched synthetic personas and digital twins, and almost nobody could say how to measure whether they work. The demos are good. The question of what they're good for is still open, and the industry is answering it in the wrong order.

Primary research stays with humans. A synthetic is not a respondent. It is a way to use research that has already been done: a persona or a simulation grounded in what those people said, so a team can act on it. Treating the two as substitutes is the wrong comparison. Saeideh Bakhshi makes the same split. She writes that the field is "so focused on making AI impersonate research participants" that it is missing a more useful kind of simulation.

Researchers who question digital twins and synthetic personas are right to. The products do not do what the demos imply, and the people who run studies can see that. They are ahead of the technology on this question, not behind it. The simulations worth building are the other kind. Frontier labs already run them: an environment, a judge, and a human dataset underneath both.

What synthetic users, digital twins, and simulations mean

"Synthetic users" covers three different things, and the argument gets muddy when they're mixed.

A synthetic user, also sold as a synthetic persona or synthetic respondent, is a language model playing a type of customer. You describe the segment, the model answers in character. What you get is the model's average view of that group, drawn from everything it read on the internet, not from your customers.

A digital twin is a language model playing one specific person. It's built from something real, usually an interview, sometimes past survey answers and CRM data. Ask it a question and it answers the way that person did, with a quote from the interview attached.

A simulation is different in kind. Nobody plays the customer. You take the situations your customers bring, the goals, the contexts, the edge cases, and run your product through them. Then something with a ground truth grades the result.

The first two put a model in the respondent's seat. The third keeps real people in that seat and uses the model to generate the situations they judge.

The first two are about to become a lot more common. In September, Salesforce agreed to acquire Listen Labs, one of the strongest AI interview platforms in the market, for a price reported near $2 billion, and described it as a customer research and human simulation platform. Listen Labs launched its twins product in October. Qualtrics, Outset, Knit, and Conveo sell versions of the same idea, and our platform comparison keeps track of which is which. When a company with Salesforce's distribution puts it behind simulated respondents, many teams will be offered them as a default, and that's a good moment to be precise about what they can and can't do.

Investors are funding that bet. Redpoint led an $80 million Series A for Aaru in April 2026. Aaru builds synthetic populations so a company can test a decision without recruiting anyone. Simile, founded by Joon Sung Park, the Stanford researcher behind the generative-agents study below, raised a $100 million Series A led by Index Ventures in February 2026 and a $200 million Series B at a $2 billion valuation five months later.

Where digital twins hold up

The skeptic's reflex, that AI can't replace people, is only half right, so twins deserve their due.

A twin is excellent at repeating what the person already said. If a shopper told your interviewer she won't buy jeans she can't return, her twin will tell you that too, in her words, every time you ask. That's useful. Research gets filed and forgotten, and a twin keeps the interview alive in a form a PM can question at 11pm.

Twins are also decent at the adjacent question. If she said she hates return fees, the twin will probably guess she dislikes restocking fees. Language models are good at that kind of nearest-neighbor inference, and with a hundred twins the guesses average into something plausible.

And twins are good at drafting. Pointing a twin at a discussion guide before fieldwork catches confusing questions faster than a pilot session does. Listen Labs' launch note draws a sensible line of its own: ask the twins for everyday decisions, and run a full study for go/no-go decisions. I'd draw it in the same place.

Where digital twins fall apart

The problem is what sits on the other side of that line.

A twin is built from the past. It knows what the person said about things that existed when you interviewed them. The decisions you most need research for are about things that don't exist yet: the feature you're about to build, the price you're about to test, the ranking change nobody has seen. Ask a twin about those and you don't get the person's judgment. You get the model's best guess, wearing the person's vocabulary, with a confidence score on it.

The evidence lines up with that. A Stanford team built generative agents from two-hour interviews with 1,052 people. On survey questions held out of the interviews, the agents matched the participants' own answers about 85% as well as the participants matched themselves two weeks later. That's the replay case, and it's a real result, with a gloss from a Great Question researcher worth keeping: in raw terms, the agents were wrong about one time in three. The same literature shows what happens when the question moves off the interview. Twins fine-tuned on a person's survey history hit a correlation of 0.98 backfilling answers the person had already given, and 0.68 on new questions. A Columbia team trained twins on 500-plus answers from each of 2,000 people, ran 19 pre-registered studies, and found an average correlation of 0.20 between twin and human, "only modestly more accurate" than a base model with no personal data at all. Pew Research built twins of its own panelists from their demographics and prior answers and ran them through three real survey waves; they missed by about 12 points on average across nearly 300 questions, and by more on the groups a brand most needs to hear from. Willingness-to-pay estimates come out wrong-signed in some cases, and the spread of opinion collapses toward one answer per group. A systematic review of 182 studies, out in March, concluded that the fidelity gains "remain modest" and recommended treating synthetic participants as heuristics.

The accuracy figures now circulating, 93% distribution accuracy against real humans from one vendor and 93% average parity from another, raise the question I keep asking: 93% accurate at what? Predicting how people answer rating-scale questions withheld from their original interview, or predicting how they'll respond to a marketing campaign they've never seen? Those are completely different things. As far as the methods are described, the figures come from the first kind of test: hold back part of an interview, then check the twin against the person's real answers, which is the replay case again. On the questions you'd pay to answer, the ones with no past to replay, nobody has published a number yet.

Some twin products decline to answer when the interview gives them too little signal, and say "I don't know" instead of guessing. That's the right design. It also marks where the approach ends, because the novel question is the one the product declines.

Underneath all of it is the ground-truth problem. When a twin is wrong, nothing tells you. A real participant can be re-contacted, re-screened, asked why. A twin's answer is self-certified. You can audit the quote it cites, but the quote is from the interview, and the question wasn't.

What happened when we built digital twins

We built digital twins at Voicepanel about a year ago and ran into all of this firsthand.

The twins were excellent at reproducing what people had already told us. When we introduced genuinely new questions or concepts, confidence dropped dramatically. When we pushed them to answer anyway, they fell back on the underlying model's assumptions rather than anything specific to the person. The result wasn't dramatically more useful than querying the original research, so we stopped.

Product teams have been here before. For twenty years we pinned personas to the wall: a name, a photo, an age, three bullet points about what she wants. Everyone knew they were a confident average of what the team already believed, and everyone used them anyway, because a face is easier to argue about than a spreadsheet.

A twin is that poster with a chat window. It's more vivid, it cites its sources, and it talks back. It's still a fixed snapshot of a person, consulted about a future it hasn't seen.

How the labs train a judge

The labs with the most to gain from a fake expert do not use one. They pay real people to say what good looks like, then train a model to apply that standard.

The method is reinforcement learning from human feedback, and the later version that puts a language model in the rater's seat. People rate a sample of outputs. Those ratings train a reward model, or they become the answer key for an LLM judge. The lab runs the system inside an environment, scores each attempt, and updates the model toward higher scores. InstructGPT is the published version of the first half: human rankings, a reward model, then a policy trained against that model. The judge only works because the humans came first.

Mercor pays doctors, lawyers, and engineers a median of $95 an hour to grade model outputs for the top labs, because the labs have concluded that measuring what good looks like is their primary bottleneck. The labs could generate a synthetic oncologist for free. They pay the real one, because the real one is the only ground truth available.

The output of that loop is a dataset, and most research teams do not produce one. A report tells a team what the study found. A dataset of scenarios, ratings, and reasons is something a builder or an agent can train against and check. The labs hand the second thing to the people who ship the model. Most research still hands over the first.

Simulate the scenario, not the person

Companies need to predict how a target population will respond. They don't necessarily need to recreate individual people. Digital twins are one approach to that prediction, and far from the only one. Frontier labs and big tech companies routinely use automated graders to evaluate AI agents, guide reinforcement learning, and improve products. Those graders can be provably calibrated to the preferences of real people without simulating a single individual.

Judgment is the expensive part of research, and scenarios are the cheap part. A language model is genuinely good at generating scenarios: hundreds of intents, contexts, follow-ups, and edge cases, far more than you'd ever write by hand. Give it a seed of real sessions and it will produce a representative sample of everything your users come to do. That's a simulation worth running.

What it should never produce is the verdict. The verdict comes from people, once, with structure: real users who match your customers, recorded on the scenarios that matter, rating what they saw against a rubric and saying why. From their ratings and their reasons you build the rubric's anchors in their words, and from the rubric you calibrate a judge. Then the judge does what people never could: it grades hundreds of simulated scenarios on every release, at machine speed, and you can measure how often it agrees with your users because you have their answers.

Humans once, to establish what good means. Judges always, to measure it. I've written up how that loop runs for AI products. A twin inverts it: it spends the human on the interview and then asks the model to be the human forever. A calibrated judge spends the human on the judgment and asks the model to apply it. The twin's answers degrade with every new question, while the judge's answers get checked against people on a cadence.

The benchmark that matters for any prediction system, twin or judge, is how well it correlates with people's responses to entirely new campaigns, concepts, or products that weren't part of its training data. That varies enormously by context, and it's the number I'd like every vendor in this category to publish.

Hamel Husain and Shreya Shankar allow synthetic traces when there's no alternative, and they say plainly, "We recommend getting real users instead of relying on synthetic data where possible." Their FAQ goes further: "synthetic data should be your last resort." I'd add one rule: use synthetic data for the inputs and keep it away from the answer key.

The same problem shows up in the simulated users AI teams use to test their own chatbots. A Carnegie Mellon study ran 31 LLM user simulators against 451 real people on the same tasks and found the simulators "excessively cooperative", giving "uniformly more positive feedback" and creating an easy mode that pushed agent pass rates above the human baseline. Simulated inputs widen the test, and simulated verdicts flatter the product.

Where synthetic data belongs in customer research

• Generating scenarios. Expand twenty real intents into two hundred, and add the contexts and edge cases your team wouldn't think of.
• Stress-testing the instrument. Run a discussion guide or a screener past a model before fieldwork.
• Red-teaming inputs. Adversarial prompts, odd phrasings, the long tail.
• Searching what people already told you. A twin is a good interface over past research, as long as it's labeled that way.
• Handing builders a dataset. Scenarios, ratings, and reasons, in a form an agent can use. A simulation should be built on that dataset, and the dataset only exists if the research was done with people.
• Deciding what's good. This is the one job a model can't do for you, and it's the job simulated respondents are usually sold for.

What research with real people looks like now

The usual objection is that real people are slow and expensive, and that gap is what simulated respondents are meant to fill. The gap is smaller than it was. On Voicepanel, one study can combine an AI-moderated interview, a survey, a usability task on a website, prototype, or mobile app, and a side-by-side comparison of concepts with blind randomization. You can recruit from more than ten panel partners, from your own customers, or from your website, in 37 languages, and screen with open-ended questions that are evaluated against criteria you write in plain English. You can design the study in the app or from your own agents through the MCP server and REST API, and pull the results the same way. The study design reference lists everything the platform can do.

That's the real-people half of the loop, built so that running it isn't the slow step.

Five questions to ask a vendor selling synthetic users

Most research teams haven't decided what they think. In User Interviews' May survey of 150 researchers, 63% said their organization has no stance on synthetic users, and 47% said they're skeptical and want more evidence. These are the five questions I'd ask.

1. What is the ground truth? If the twin is wrong, how would I find out, and when?
2. How is accuracy defined, and on which questions? Answers held out of the interview, or concepts the person never saw?
3. What happens on a concept the person never saw? Show me a case where the twin said something the real person later contradicted.
4. Can I see the disagreement? Not the confidence score. The measured gap between twins and the humans they're modeled on, by question type.
5. What does it cost per answer, and what does it cost per real person I could have asked instead?

A vendor with good answers to all five is selling you a search interface over real research, and that's a fine thing to buy. A vendor who can only answer with a confidence score is selling you the persona poster.

What comes next is public benchmarks: predictive accuracy on new concepts, measured the same way across twins, synthetic panels, and calibrated judges. Until those exist, the method worth copying is the one the labs use. Sample the situations, pay real people to judge them, and calibrate everything else against those judgments. Simulate the scenario rather than the person.

Synthetic users FAQ

What are synthetic users?
Synthetic users, also sold as synthetic personas or synthetic respondents, are language models prompted to answer research questions as a type of customer. They answer from the model's general knowledge of that group, not from your customers.

What is the difference between synthetic users and digital twins?
A synthetic user plays a type of person. A digital twin plays one specific person, built from that person's interview or survey history, and cites it. Both put a model in the respondent's seat. A simulation keeps real people there and uses the model to generate the situations they judge.

Are synthetic personas and digital twins accurate?
On questions a person has already answered, twins replay them well: Stanford's generative agents matched held-out survey answers about 85% as well as the participants matched themselves two weeks later. On new questions the published correlations fall, to 0.68 in one study and 0.20 in another, and Pew's twins missed by about 12 points on average. No vendor has published accuracy on concepts the person never saw.

What are the alternatives to synthetic users?
Simulate the scenario instead of the person: use a model to generate the situations your users bring, have real users judge a sample once with structure, and calibrate an automated judge against them. For the research itself, AI-moderated interviews, surveys, and usability tasks now run with hundreds of real people at once, which removes most of the cost case for a stand-in.

Can digital twins replace customer interviews?
Not for decisions about things your customers haven't seen, which are most of the decisions research exists for. They work as a searchable record of interviews you've already run, and the vendors' own guidance says to run a full study for go/no-go decisions.

Grade runs this loop for AI product teams: we recruit your target users, record them on the scenarios that matter, help you build a rubric from what they say and do, and deliver an autograder calibrated to them. Join the waitlist. For every other kind of customer research, start with the study design reference or talk to us.