After reading this article from Nielsen about the size of usability studies [1] their research credibility has gone down for me. According to the article, no matter the size of a user base:
"... the answer is simple: test 5 users in a usability study. Testing with 5 people lets you find almost as many usability problems as you'd find using many more test participants."
Oh wait, but my site has millions of users!
"Doesn't matter for the sample size, even if you were doing statistics."
Even if you were doing statistics? What does it even mean lol... it just sounds to me they want to justify whatever "studies" they've have in their library. I'd say, unless a Nielsen study is replicated and double checked, probably (pun intended :-p) their studies shouldn't be taken very seriously.
The point the article you're quoting is making is that if you insist on a single number then 5 is enough if your question falls in a specific category. Namely, for the purpose of finding the number of people needed to get the biggest payoff in terms of number of usability problems identified.
This being the key point they're making:
> The main argument for small tests is simply return on investment: testing costs increase with each additional study participant, yet the number of findings quickly reaches the point of diminishing returns. There's little additional benefit to running more than 5 people through the same study; ROI drops like a stone with a bigger N.
This doesn't quantify the scale of the reported problems. It goes into it with the assumption that you simply want to find as many of the problems with your site as possible.
> Even if you were doing statistics?
The sentence you quoted regarding this makes the point that the number of users the site has is not the factor that should drive the size of your panel (you'll note that elsewhere the article argues that where you need statistically significant results you need at least 20 users and more if you need tight confidence intervals). Rather the size of your panel should be determined by your required confidence intervals, and when identifying usability issues is your main concern they don't need to be tight.
"... the answer is simple: test 5 users in a usability study. Testing with 5 people lets you find almost as many usability problems as you'd find using many more test participants."
Oh wait, but my site has millions of users!
"Doesn't matter for the sample size, even if you were doing statistics."
Even if you were doing statistics? What does it even mean lol... it just sounds to me they want to justify whatever "studies" they've have in their library. I'd say, unless a Nielsen study is replicated and double checked, probably (pun intended :-p) their studies shouldn't be taken very seriously.
1: https://www.nngroup.com/articles/how-many-test-users/