AI Respondents Matched Humans' Answer but Flattened Its Variety

When Nielsen Norman Group posed one open-ended survey question to 120 AI-generated respondents, the synthetic group reached the same overall conclusion as 604 real tech and design professionals — but with far fewer distinct answers.

That's the headline finding from the firm's latest experiment with so-called synthetic users, generative-AI stand-ins for research participants. The study, run in February 2026 on the Synthetic Users platform, took a narrower scope than the team's earlier work on AI participants in interviews. This time the task was tightly scoped: answer a single open-ended question lifted from an existing survey.

The question came from NN/g's earlier survey on UX terminology, in which 604 professionals said what name they would use for UX work. The researchers fed the identical prompt to 120 simulated respondents, then compared the two datasets side by side.

What did the comparison show?

Both datasets pointed the same way. Whatever conclusion you'd draw from the human survey, the synthetic sample supports it too.

The difference lies in the spread. Human respondents produced a wider range of distinct answers; the synthetic ones clustered tightly, generating much less variation. If you're using AI respondents to map the full landscape of opinions — the long tail of unusual naming preferences, say — this flattening matters.

Why does variation matter for survey research?

Open-ended questions exist precisely to surface diversity of thought. A tool that reproduces the modal answer while trimming the outliers will still look valid on a top-line read. It will quietly fail at the exploratory job surveys often do.

Think of it this way: two datasets can agree on the winner while disagreeing completely on how contested the race was.

What's the practical takeaway?

For teams weighing synthetic users against recruitment, the study suggests a split verdict:

• Same directional conclusion as the human sample
• Far fewer distinct answers among AI-generated responses
• Results generated in February 2026 via the Synthetic Users simulation platform

That makes synthetic respondents a plausible smoke test before committing budget to human fieldwork, and a risky substitute for it — at least where the richness of individual answers carries the insight.

NN/g says it has already tested synthetic users in interview settings; this survey-question experiment extends that line of inquiry. Expect the firm to keep probing where the AI stand-ins hold up and where they collapse, as adoption of simulated participants grows across UX research teams.