Understanding Statistical Significance

Risks of Quantitative Studies

<aside> 📌 In UX research, most of the time the answer is start at n=5.

</aside>

• The most important thing about the number of users, is knowing why you’re choosing to test with that many, and then using the insights responsibly.
• n=5 is completely valid for qualitative usability studies. But you should NOT use any quantitative data from these studies as representative of all users and avoid reporting statistics (percentages, averages, etc.) in your readouts.
• Q: Can you collect quantitative metrics in a qualitative research method (ex. during usability studies)? A: Yes and no. Read more here

• How many test users in a usability study - Qualitative Usability Testing Guide | | Quantitative Usability metrics | n=20-40 | !! **This number is not for all quantitative research, but for usability metrics like time on task | - Quantitative studies: How many users to test
• 40 users
• Risk of quantitative studies | | Card Sort | n=15 | 15 users = .9 20 users = .93 30 users = .95 | - Card sorting: How many users to test | | Eye-tracking | (n=39) | n=39 for heatmaps (n=30 → R^2 = .85) n=6 for gaze replays | - How many test users
• Eye-tracking study report - How to conduct eye tracking studies | | Contextual inquiry | n=4-30 | Small: 4-12 Med: 12-24 Large: 25-30 | https://www.nngroup.com/videos/how-many-users-contextual-inquiry/ | | Discovery | n=5-30 | Start by interviewing 5–30 users (as a rolling sample of 5 users per group, until you uncover only a few new insights with each new interview | | | Personas | n=5-30 | Start by interviewing 5–30 users (as a rolling sample of 5 users per group, until you uncover only a few new insights with each new interview | | | Quantitative research | n=80 | Quantitative user testing, with 9% margin of error (50% of the time, result is within 4% of true mean) **** | https://www.nngroup.com/articles/quantitative-studies-how-many-users/ |