Jakob Nielsen Turns UX Research Methods on AI Progress

Jakob Nielsen, the usability researcher who helped found the discipline of user experience, has published an argument that AI progress can be estimated with the same design and research methods practitioners already use to evaluate software.

The piece, titled "Estimating AI's Rate of Progress with UX Design and Research Methods," appeared on his newsletter, Jakob Nielsen on UX. Its premise is direct: the tools of empirical UX work — observation, task-based testing, measurement against user goals — can tell us how fast AI systems are actually improving, independent of vendor claims.

Why does this matter for practitioners?

Nielsen has spent his career arguing that what matters about a system is not its internal sophistication but whether people can accomplish tasks with it. Applied to AI, that standard shifts attention away from benchmark scores and model parameters toward observable performance on real tasks performed by real users.

For designers and researchers, this reframes AI tools as interfaces under test rather than as breakthroughs to be admired. The methods do not change: define a task, recruit users, observe, measure success and time-on-task. What changes is the object of measurement — a system whose behavior varies run to run.

What does the UX lens add to AI evaluation?

Nielsen's long-standing position, argued across decades at Nielsen Norman Group, holds that small samples of users reveal most severe usability problems. The same logic, applied to AI, suggests that structured observation of a handful of realistic sessions can expose where current systems fail, without the cost of large-scale benchmarking.

This places a familiar instrument in unfamiliar territory. Standard technology evaluations measure whether a capability exists. UX research measures whether a person can use it to finish something. Nielsen's proposal is that the second measurement is the honest one for judging progress, because it captures the gap between what a model can do in principle and what a user gets done in practice.

The timing is deliberate. AI vendors announce capability gains at a rapid cadence, and practitioners face the practical question of when a given tool becomes reliable enough to design into a workflow. A research-based estimate of the rate of progress — observed, repeated, task-anchored — gives teams a defensible basis for that decision instead of a press release.

How might teams act on it?

The implication for design and research teams is that they already own the evaluation toolkit. Usability testing, heuristic evaluation, and longitudinal observation transfer to AI products with modest adaptation, and they produce evidence in the currency practitioners understand: task completion, error rates, and user effort.

Nielsen's contribution here is not a new method but a reassignment of an old one — pointing established UX practice at the question everyone is asking about AI and few are answering empirically.