When two people arguing about a study, I often hear one of them talk about how a study automatically flawed and can’t be trusted. However, studies with small sample sizes regularly appear in meta-analyses. Why aren’t they automatically considered useless?
In: Mathematics
TL;DR there are statical tests that can help mitigate the small samplet to a degree.
There are some topics like the one I am working on. Does Bavioral Activation compared to treatment as usual have an affect on PTSD symptoms in veterans with PTSD? Veterans is already a small population and then when you add PTSD it gets even smaller but the results are no less meaningful.
A study is only useful if its representative of the population it is meant to represent. Would you respect a study about videogame violence if all the respondents were 50 year old moms? Would you trust a new medical device thats only tested on 5 people? What if you had some weird genetics that those 5 didnt and you get hurt as a result? It is hard to represent a large population with only a few people because we are each unique in many different ways (such as gender, sexual orientation, political views, weight, height).
Say you have balls in a bag of many different colors and your goal is to grab with your eyes closed one of each color. Say there are 7 colors in total. You grab 5 balls and you get 3 red and 2 green. You see that this is not representative of the whole population which is 7 colors, so a study with this sample will only be relevant toward red and green balls. Say you increase your sample to 15 and get red, green, and blue balls. This is better than before but still not very reflective of the overall population.
The more you increase the size of the sample the better chance you have of equally representing the 7 colors in the proportion they appear in the bag.
The “best” sample size is theoretically the entire population (so a study about students will ideally need to include all the students in the world). But this is impossible so we settle with smaller samples because thats easier and cheaper to work with
Meta analysis takes many studies and combines them, and in effect combines all of their sample sizes. Do note that in meta analysis, the sample size of each study included in it plays a role in how much influence it has on the final result. So a study with 5 people in it will contribute less than one with 500.
Because you can measure how “truthful” a study is with something called statistical significance.
Let’s say you suspect your six sided die is loaded to always land on six. You throw it three times and get three sixes. Does that mean the die is loaded? Well not necessarily, because you absolutely can get three sixes in a row with a normal die too.
So you throw it ten times and get eight sixes and two fives. NOW are you sure the die is loaded? You’re far more sure than after rolling it three times, but there is STILL a possibility to get these results naturally with an unloaded die – it’s just extremely unlikely.
How “sure” you are that the results are because of the measured factor rather than random chance is “statistical significance”.
Scientists typically aim at statistical significance level of at least 95%. The exact value is always in the study, or even in the abstract. Generally speaking the statistical significance is one of the easiest things to check because it’s just high school level math. You can be certain that every published study is statistically significant, i.e. that the sample size is “big enough” – if it wasn’t, it wouldn’t be published.
Which is why dismissing studies because they have “small sample sizes” is a very amateur move and done mostly by people who are desperate to deny their results.
In observational analysis, there are many ways to approach small sample sizes while producing a useful result. The widely-used t-distribution is explicitly made for dealing with double-digit sample sizes under ~70 where using the standard normal distribution tends to be unhelpful. One might also try quasi-experimental methods — matching, propensity scores, regression discontinuity design, synthetic control, etc. — in which one attempts to coerce the statistical underpinnings of experimental analysis on a dataset *post-hoc*.
In *actual* experiments, sample size is still important, but the size needed for conventionally-useful findings varies widely by discipline and subject of interest. Social science, for example, has much less stringent conventions concerning the statistical significance of hypothesis testing results (i.e. the p < .05, or “95% chance the true population effect we estimated is inside the confidence interval for our sample”, which for most things is interpreted as “a 1/20 chance that we observed something due to randomness rather than a systemic process”) compared to something like high-energy particle physics (i.e. the infamous five-sigma criterion, or “roughly 1 in 3.5 million chance the true effect is *not* in our CI”).
There is also an epistemic/ontological (and practical, besides) concern discussed quite a bit by the statistics community in recent years regarding the tendency of applied disciplines — especially social sciences — to rely on hard cutoffs for results to be considered meaningful (e.g. the p < 0.05 above). [This](https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1583913) piece from Wasserstein et al. in 2019 gives a good and candid discussion of this issue and how we might work to reframe hypothesis testing.
So, depending on context, disciplinary standards, and what statistical tools are amenable to the data, you can still extract useful information from small samples.
One of the purposes of a meta-analysis is to aggregate studies together in order to increase the effective sample size. Meta-analyses will skew towards small sample sizes because large studies benefit less from it.
Also, most people do not understand the research process. Studies are expensive, smaller, weaker studies are less about proving something, and more about justifying the expense of conducting a more conclusive study.
Latest Answers