The Small N

Recently I had a revelation, a moment of inspiration. Then again, maybe I am just crazy.

Say a hypothetical organization collects data about the climate and the work environment within the ranks of their employees. Say also that the data shows some serious discrepancy between members of the larger group (majority) and members of the minority (underrepresented group). This hypothetical organization excuses these as saying "but we can trust this because the N is small."

Now call me crazy, but isn't that the definition of a minority? A small N? If that is the excuse to ignore it, why do the survey in the first place? Is this hypothetical organization just trying to find statistical significance within the majority only? Because, they will NEVER find a large N among the minority group. That will only happen the day when the minority becomes (or gets close enough) the majority group.

Said hypothetical organization needs to stop looking for statistical significance. They are NEVER going to find it. That is not the point of these studies. This is not new. When you do product testing, the incidence of a defect on the product is an indication that there are other defects. This is why in usability testing (my own reasearch area) we test things with just 5 people. We are not out to prove that the technology is perfect. We are out to find that something is wrong. Finding one defect is enough to say "We have work to do."

Similarly, these climate surveys that a hypothetical organization conducts should not be conducted to prove statistically that we are doing the right thing. Instead, the goal should be to identify things left to do. And there are always things left to do. Particularly for that small N.

What do you think? This is a life altering revelation or am I crazy?

Posted on 09/14/2009





Recent blogs