UX research for startups: how many people you need depends on the problem you must catch

The best-known rule in UX research says that five users find about 85% of usability problems. It is true, as an average, for problems that hit about one user in three. For a problem that hits one person in ten, five sessions show it only 41% of the time. So the useful question is not "how many users", but what is the rarest problem I cannot afford to miss. Everything below follows from that: sample sizes for tests and interviews, how to ask without leading, how to rank what you find, and what the numbers can show once you have launched.

What five users actually promises

The model behind the rule says the share of problems found is 1 minus (1 minus p) to the power n, where p is the chance that any one user runs into a given problem and n is the number of users. Nielsen and Landauer (1993) put the typical p at 31%, averaged across the projects they studied, and Nielsen's write-up for Nielsen Norman Group (NN/g) adds that the formula holds only for comparable users using the site in fairly similar ways. Five users and p of 31% is where 85% comes from. We worked out the same formula for other values of p; MeasuringU's published table agrees (41% at five users and p of 10%).

Read the table by row. Five sessions will almost always show a problem that hits half your users, and will miss most problems that hit one in ten. Real data is less tidy than the average:

• Spool and Schroeder (2001) tested four shopping sites. On one, the first five participants found 35% of the problems, new problems kept appearing through all 18 participants, and serious ones that stopped a purchase surfaced only with the 13th and 15th. We saw this only through secondary summaries.
• Faulkner (2003) drew random groups from 60 real users of one application. Groups of five found anywhere from 55% to 99% of the problems, averaging about 85%. At ten users the worst group found 80%, and at twenty, 95%. The average held, but a single group of five could find half.
• Lindgaard and Chattratichart (2007) analysed nine commercial teams from the CUE-4 comparison and argued that covering more tasks mattered more than adding users. We read only their summary.

Sources disagree, and the way to reconcile them is to treat the five-user rule as a statement about common problems. What follows from it:

• Cover tasks before adding people. Five users on three tasks see a thin slice of the product.
• Test each distinct group separately. NN/g suggests three or four users per group for two groups, and three per group for more.
• Test in rounds. NN/g prefers three studies of five, with fixes between them, to one study of fifteen.
• Do not sample for rare disasters. A payment that fails for one customer in fifty is unlikely to turn up in a test. Log errors and watch for them in analytics.
• Steve Krug's version is the smallest that still works: about three participants, one morning a month, a debrief over lunch. He concedes it is not statistically valid, and argues it still shows the worst problems.

Interviews: know what you are saturating

NN/g says there is no golden number for interviews. Start with five or six and add people until new ones stop teaching you anything. Its examples of published figures run from 5 to 50: Griffin and Hauser estimated 20 to 30 interviews for customer needs, and Guest, Bunce and Johnson found saturation by 12.

The most useful finding is that saturation has two meanings. Hennink, Kaiser and Marconi (2016) analysed 25 interviews from one study and found that code saturation, the point where you have heard each theme, came at nine interviews. Meaning saturation, where you understand why well enough to rely on it, took 16 to 24. For a startup that is a design rule: nine interviews tell you what the problems are; designing from the reasons needs about twice that. Both results come from fairly homogeneous groups, so two customer segments mean two rounds.

A stopping rule that works in practice (ours, not from those papers): after every interview, add what was new to a running list, and stop when two interviews in a row add nothing.

For the questions, the Mom Test (Rob Fitzpatrick) gives the rules most startups need: talk about the person's life, not your idea; ask about specific past events; and treat three kinds of fluff as noise, which are generic claims ("I always"), future promises ("I would") and hypotheticals ("I might"). A meeting that ends with a next step, such as time, an introduction or money, is evidence. A compliment is not. NN/g gives the measured reason: across studies, stated preference correlated with measured performance at 0.44 for PC applications and 0.53 for websites. Squared, that is about 19% and 28% of the variation, so liking predicts a fifth to a quarter of how well something works.

A six-question script:

1. Tell me about the last time you [did the job]. What happened?
2. What did you do first? Then what?
3. What was the hardest part, and what did you do about it?
4. What did you already try to fix it, and what did that cost you in time or money?
5. Who else had a say in how it got done?
6. Who else should I talk to?

Write tasks that do not name the interface

NN/g's rules for task scenarios are to make them realistic, make them something to do and not to describe, and avoid clues. GOV.UK's research team calls the failure trigger words: words on the page that steer people, such as "Can you add something to your basket?". Their neutral version is "Show me how you buy the shirt." Here is the same fix applied to two published LAIN designs. These are illustrations of task wording, not accounts of research we ran.

Talk less than you want to. NN/g describes three moves. The echo repeats the participant's last words with a questioning tone, using their exact words. The boomerang answers a question with a neutral one, such as "What would you normally do?". The Columbo asks half a question and trails off. Do not answer "Do I need to register?" with "No, that's fine".

On thinking aloud, evidence is mixed. Van den Haak, de Jong and Schellens found that thinking aloud while working hurt task performance on a library catalogue, while a University of East Anglia study of 30 participants found no effect on performance, and the concurrent group reported more interface problems. So think aloud to find problems, and do not time tasks while people do it.

Rate severity before you fix anything

NN/g, following Nielsen, sets severity from three factors: how often it happens, how hard it is to get past, and whether it persists after people learn it. A single rater is unreliable, and the mean of three is acceptable. A rubric for a small team, built on those factors (the thresholds are our judgement):

Raise a problem one level if someone hits it again after learning it. Two people score separately, take the higher number when they differ by one, and talk when they differ by two.

After launch: what small numbers can and cannot show

Once real people use the product, three standard calculations tell you how much to trust a count. They are ours, from the usual formulas.

The A/B figures assume a two-sided test at 5% significance, 80% power and an even split; a simulation of 20% against 30% and 20% against 24% gave 82% and 79% power. At 200 sign-ups a week, 5% to 6% would take about 82 weeks. Small funnels are just as loose: if 1 of 5 people reach a step, the true rate is anywhere from 4% to 62% (95% Wilson interval). If 0 of 10 hit a bug, it could still affect up to 28% of people. Below a few thousand visitors a month, use analytics to choose where to watch people, and use sessions to learn why.

Where it goes wrong

A rhythm for a small team

How this fits a project

The first LAIN phase, Alignment, is about learning your goals, brand and audience before any screen exists. Findings from interviews or tests you have already run make that phase sharper. An app or dashboard design project takes 8 to 12 weeks, and a clickable prototype is the natural thing to put in front of five people before development. The Omni Agent, Aerium and Lima AI pages show the kind of interface those sessions would use. To go on from here, read UI vs UX design and onboarding UX design. If you are about to design a product and want help choosing what to test, start a project.